License plate reading system with enhancements

ABSTRACT

System and methods are disclosed for capturing license plate (LP) information of a vehicle in relative motion to a camera device. In one example, the camera system detects the LP in multiple frames, then aligns and geometrically rectifies the image of the LP by scaling, warping, rotating, and/or performing other functions on the images. The camera system may optimize capturing of the LP information by executing a temporal noise filter on the aligned, geometrically rectified images to generate a composite image of the LP for optical character recognition. In some examples, the camera device may include an image sensor, such as a high dynamic range (HDR) sensor, modified to set long and short exposures of the HDR sensor to capture frames of a vehicle&#39;s LP, but without consolidating the images into a composite image. The camera system may set optimal exposure settings based on detected relative speed of the vehicle.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. National Phase of InternationalApplication No. PCT/US2020/030789, filed on Apr. 30, 2020, designatingthe United States of America and claiming priority to U.S. patentapplication Ser. No. 16/399,607, filed on Apr. 30, 2019, U.S. patentapplication Ser. No. 16/399,654, filed on Apr. 30, 2019, and U.S. PatentApplication No. 62/841,060, filed on Apr. 30, 2019. This applicationclaims priority to and the benefit of the above-identified applications,which are fully incorporated by reference herein in their entirety.

This application claims the benefit of priority from U.S. patentapplication Ser. No. 16/399,607, filed Apr. 30, 2019, which wasincorporated by reference in the aforementioned U.S. Provisional PatentApplication Ser. No. 62/841,060, and which is again herein incorporatedby reference in its entirety.

This application claims the benefit of priority from U.S. patentapplication Ser. No. 16/399,654, filed Apr. 30, 2019, which is hereinincorporated by reference in its entirety.

BACKGROUND

Video and still cameras affixed to stationary structures are sometimesused for purposes of security surveillance. In a stationaryinstallation, the camera is typically in an environment with knownexternal variables (e.g., environmental, lighting, field of view) thatare generally constant or readily apparent. In such an environment,basic cameras with minimal enhancements might suffice.

Meanwhile, in police cars, taxis, crowdsourced ride-sharing vehicles,and even personal vehicles, cameras mounted on a dashboard are sometimesused for purposes of recording the environment in the immediateproximity of the vehicle. However, in a vehicle moving at high speeds,the capabilities of a traditional camera to capture video and stillimages can sometimes be compromised. Moreover, external variables cansometimes further negatively impact the ability for the camera tocapture sharp, useful images.

With respect to lighting conditions, some security cameras includefeatures to improve recordability in low-light scenarios and night time.In the case of a stationary camera installation, a separate light sourcewith a daylight sensor and/or clock setting might be installed in thearea to illuminate in low-light scenarios or at night. Moreover, someseparate light sources might emit light in the infrared spectrum rangeto enhance recordability at night without necessarily illuminating theenvironment with visible light. One problem is that in low lightconditions, images of license plates tend to be very noisy, and it canbe difficult/impossible to accurately detect the characters in a licenseplate. Long exposure times cannot be used to solve this problem becausewhen the license plate is in motion, the captured image would beblurred.

Another problem is that incoming vehicle traffic and following vehicletraffic are both in motion, and likely with different speeds relative toa subject vehicle (i.e., the camera car). Thus, at a given exposuresetting, some portions of a captured image may be higher quality thanothers, and these portions may vary from frame to frame. In addition,any angular motion relative to the subject vehicle might result in thelicense plate being captured in a shape other than a perfect rectangle.This further complicates the ability to recognize the characters in thelicense plate.

Yet another shortcoming is that incoming vehicle traffic and followingvehicle traffic are moving at different speeds relative to a subject car(i.e., the camera car). And, a single camera cannot accurately captureboth vehicles with a single exposure setting—historically, a singleimage sensor is unsuitable to simultaneously set two different exposureto capture them both.

Numerous novel and nonobvious features are disclosed herein foraddressing one or more of the aforementioned shortcomings in the art.

BRIEF SUMMARY

In light of the foregoing background, the following presents asimplified summary of the present disclosure in order to provide a basicunderstanding of some aspects of the embodiments disclosed herein. Thissummary is not an extensive overview of the invention. It is notintended to identify key or critical elements of the invention or todelineate the scope of the invention. The following summary merelypresents some concepts of the invention in a simplified form as aprelude to the more detailed description provided below.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. Onegeneral aspect involves a license plate recognition (LPR) systemattached to a law enforcement vehicle (or other vehicle). The LPR systemmay include a camera device including an image sensor, the camera deviceis configured to capture images including long-exposure images andshort-exposure images with the image sensor, and the image sensor isconfigured to nearly simultaneously output a long-exposure image of afield of view and a short-exposure image of the same field of view. Inother words, the same image may be captured, but with different exposure(or other) settings on the image sensor and/or camera device. The LPRsystem may also include a computer memory configured to store the imagesoutputted by the image sensor, and a processor, which is communicativelycoupled to the memory.

The processor may be programmed to perform steps of a method of an LPRsystem. For example, the processor may receive, from the memory, a firstlong-exposure image and a first short-exposure image. The firstlong-exposure image may be captured with a first long-exposure settingof the camera device, and the first short-exposure image may be capturedwith a first short-exposure setting of the camera device. The processorof the LPR system may also detect a first license plate and a secondlicense plate in the first long-exposure image, where the first licenseplate is in a first portion of the field of view and the second licenseplate is in a second portion of the field of view, and where the firstportion of the field of view is different than the second portion of thefield of view. The processor may also detect the first license plate andthe second license plate in the first short-exposure image, where thefirst license plate is in the first portion of the field of view and thesecond license plate is in the second portion of the field of view. TheLPR system may result in the characters of the second license platehaving a greater probability of being recognized by a computerizedoptical character recognition platform in the first short-exposure imagethan in the first long-exposure image. In some embodiments, the LPRsystem may result in the characters of the first license plate have agreater probability of being recognized by a computerized opticalcharacter recognition platform in the first long-exposure image than inthe first short-exposure image. Other embodiments of this aspect includecorresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. TheLPR system where the processor is further programmed to: calculate arelative speed of the second license plate using motion blur analysis ofthe second license plate in the first short-exposure image; capture,using the image sensor, a next short-exposure image with an exposuresetting based on the calculated relative speed to reduce motion blur.

The LPR system may further include a controller communicativelyconnected to the camera device, where the controller is configured toadjust an exposure setting of the image sensor to affect the capture ofthe long-exposure images and the short-exposure images. Moreover, theprocessor may be further programmed to instruct the controller to adjustthe first long-exposure setting and the first short-exposure setting ofthe camera device by an amount. The processor may also capture, usingthe image sensor, a second long-exposure image having a secondlong-exposure setting and a second short-exposure image having a secondshort-exposure setting. In addition, the processor may detect the firstlicense plate and the second license plate in the second short-exposureimage. The processor may also align the second license plate in thefirst short-exposure image with the second license plate in the secondshort-exposure image. The processor may then transform the secondportion of each of the first short-exposure image and the secondshort-exposure image by geometrically rectifying to accommodate forrelative positions of the second license plate. Then, merge at least thefirst short-exposure image and the second short-exposure image into aconsolidated image. The result of the LPR system may be that charactersof the second license plate have a greater probability of beingrecognized by the computerized optical character recognition platform inthe consolidated image than the first short-exposure image, the secondshort-exposure image, the first long-exposure image, or the secondlong-exposure image. The LPR system may also include examples where themerging of images into a consolidated image includes merging additionalshort-exposure images from among the images captured by the LPR system,where the additional short-exposure images include the second licenseplate in the second portion of the field of view. The LPR system mayresult, in some examples, where characters of the first license platehave a greater probability of being recognized by a computerized opticalcharacter recognition (OCR) platform in the first long-exposure imagethan in the first short-exposure image. While the preceding examplesrefer to a first license plate or a second license plate, thecontemplated embodiments are not so limited—e.g., a field of view mayinclude more than two license plates and the accuracy of the OCRing maybe different for each of the license plates based on various factorsdiscussed herein, including but not limited to the relative speed of thevehicle onto which the license plate is affixed, varying lightingconditions at different spots in the field of view, dimensions and othercharacteristics (e.g., text color, background color, typeface, and thelike) of the characters in the license plate, and other factors.

Implementations of the described techniques may include hardware, amethod or process, or computer software on a computer-accessible medium.One general aspect includes LPR systems operating in a serial mannerwhere the image sensor is nearly simultaneously outputting images of thefield of view where the setting of the camera device is alternated everyother frame from the long-exposure setting to the short-exposuresetting, where the image sensor is a single image sensor. Meanwhile,another general aspect includes LPR systems operating in a parallelmanner where the image sensor is nearly simultaneously outputting imagesof the field of view where the setting of the camera device is thelong-exposure setting for a first set of lines in a frame whilesimultaneously to the short-exposure setting for a second set of linesin the same frame, where the first set of lines is different than thesecond set of lines. Another general aspect includes LPR systemsoperating where the image sensor includes a high dynamic range (HDR)sensor, and the HDR sensor is nearly simultaneously outputting images ofthe field of view includes separately outputting the long-exposure imageand the short-exposure image without consolidating the long-exposureimage and the short-exposure image into a single, consolidated image.

Another aspect includes LPR systems where the processor is furtherprogrammed to instruct a controller to adjust, for each of a pluralityof images, at least one of a shutter speed setting, ISO setting, zoomsetting, exposure setting, and/or other settings of the camera devicesuch that a subsequent image is captured by the camera device with adifferent setting than that used to capture an immediately precedingimage.

Another general aspect includes the LPR system where the processorincludes an application specific integrated circuit (ASIC) processor,and the camera device is communicatively coupled to the processor by awired connection. Other embodiments of this aspect include correspondingcomputer systems, apparatus, and computer programs recorded on one ormore computer storage devices, each configured to perform the actions ofthe methods.

Yet another general aspect includes the LPR system where the cameradevice is a camera assembly further operating as an enclosure for theimage sensor, controller, processor, and the memory arranged therein. Insome embodiments, the camera device may include one or more of theaforementioned components. In other embodiments, the camera device mayinclude an image sensor and associated electronic circuitry, but one ormore of the other aforementioned components may be outside of the cameradevice but enclosed within a single camera assembly. In yet otherembodiments, the camera device may include an image sensor andassociated electronic circuitry, but one or more of the otheraforementioned components may be outside of the camera device andcommunicatively coupled to the camera device through one or moreinterfaces and connections, e.g., a wired connection between a cameradevice mounted near a windshield of a vehicle and a processor, which maycomprise a GPU, located in a trunk of the vehicle. Alternatives to thedevices and components described herein are possible—e.g., individualmodules/components or subsystems can be separated into additionalmodules/components or subsystems or combined into fewermodules/components or subsystems and may be interconnected through oneor more interfaces and connections.

Also disclosed herein is a method involving one or more components ofthe license plate recognition (LPR) system disclosed herein. The LPRsystem may, in some examples, include a camera device with an imagesensor (e.g., an HDR sensor or other sensor types), one or moreprocessors, one or more computer memories, and/or a controller. Themethod may include steps to receive, by the processor from the memory, afirst long-exposure image of a field of view captured by the imagesensor with a long-exposure setting and a first short-exposure image ofthe same field of view captured by the image sensor with ashort-exposure setting. In some examples, the short-exposure image andthe long-exposure image are nearly simultaneously outputted by the imagesensor. The method may further include a step to detect, by theprocessor, a first license plate and a second license plate in the firstlong-exposure image, where the first license plate is in a first portionof the field of view and the second license plate is in a second portionof the field of view. In some examples, the first portion of the fieldof view is different than the second portion of the field of view, asillustrated herein. The method may further detect, by the processor, thefirst license plate and the second license plate in the firstshort-exposure image, such that the characters of the first licenseplate have a greater probability of being recognized by a computerizedoptical character recognition platform in the first long-exposure imagethan in the first short-exposure image. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Also disclosed herein is a tangible, non-transitory computer-readablemedium or computer memory storing executable instructions that, whenexecuted by a processor of a license plate recognition (LPR) system,cause the LPR system to perform one or more of the steps of the methodsdisclosed herein. In one example, the computer-readable medium may storeexecutable instructions that, when executed by a processor of the LPRsystem, cause the LPR system to receive, from a memory of the LPRsystem, a first long-exposure image of a field of view, where the firstlong-exposure image is captured, using an image sensor, with along-exposure setting of a camera device of the LPR system; receive,from a memory of the LPR system, a first short-exposure image of thesame field of view, where the first short-exposure image is captured,using the image sensor, with a short-exposure setting of the cameradevice; detect a license plate in the first long-exposure image, wherethe license plate is in a first portion of the field of view; detect thelicense plate in the first short-exposure image, where the license plateis in the first portion of the field of view, where characters of thelicense plate have a greater probability of being recognized by acomputerized optical character recognition platform in the firstlong-exposure image than in the first short-exposure image; instruct, acontroller communicatively coupled to the processor and camera device,to adjust the long-exposure setting of the camera device by a firstamount and to adjust the short-exposure setting of the camera device bya second amount, where the long-exposure setting and short-exposuresetting include at least one of a shutter speed setting, ISO setting,zoom setting, and exposure setting of the camera device; capture, usingthe image sensor, a second long-exposure image with the adjustedlong-exposure setting and a second short-exposure image with theadjusted short-exposure setting; detect the license plate in the secondlong-exposure image and the second short-exposure image; align thelicense plate in the first long-exposure image with the license plate inthe second long-exposure image; transform the first portion of each ofthe first long-exposure image and the second long-exposure image bygeometrically rectifying to accommodate for relative positions of thelicense plate; and merge at least the first long-exposure image and thesecond long-exposure image into a consolidated image, where charactersof the license plate have a greater probability of being recognized bythe computerized optical character recognition platform in theconsolidated image than the first short-exposure image, the secondshort-exposure image, the first long-exposure image, or the secondlong-exposure image. Other embodiments of this aspect includecorresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods. For example, in some examples thefirst amount and second amount are random amounts; in other examples,they are predetermined amounts, such as predefined values or valuescalculated based on a predetermined algorithm or formula; in yet otherexamples, one or more of the amounts may be based on a calculatedrelative speed of a target license plate using motion blur analysis ofthat license plate in a previously captured image, either with along-exposure setting or short-exposure setting. Implementations of thedescribed techniques may include hardware, a method or process, orcomputer software on a computer-accessible medium.

The methods and systems of the above-referenced embodiments may alsoinclude other additional elements, steps, computer-executableinstructions or computer-readable data structures. In this regard, otherembodiments are disclosed and claimed herein as well. The details ofthese and other embodiments of the present invention are set forth inthe accompanying drawings and the description below. Other features andadvantages of the invention will be apparent from the description,drawings, and claims.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. Onegeneral aspect includes a memory and a processor programmed to performseveral operations. The memory may be configured to store image data(e.g., long-exposure images, short-exposure images, and other data)captured by a camera device attached to a police vehicle. The image dataincludes a first image of the target vehicle at a first time and asecond image of the target vehicle at a second time. The processor,which is communicatively coupled to the memory, may be programmed toperform numerous operations.

For example, the processor may receive the first image from the memory,where the first image shows the target vehicle at a first position.Moreover, the processor may detect a license plate in the first image,where the license plate is in a first portion of the first image. Inaddition, the processor may receive the second image from the memory,where the second image shows the target vehicle at a second positionthat is different from the first position. Moreover, the processor maydetect the license plate in the second image, where the license plate isin a second portion of the second image. In addition, the processor mayalign the license plate in the first portion and the license plate inthe second portion. The processor also transforms the first portion andthe second portion of the license plates by geometrically rectifying toaccommodate for relative positions of the target vehicle at the firstposition and the second position. After the transforming, the processormay execute a temporal noise filter on the first portion of the firstimage and the second portion of the second image to generate aconsolidated image, where the consolidated image has a higherprobability that characters of the license plate in the consolidateimage are recognized by a computerized optical character recognitionplatform than the license plate in the first image. Other embodiments ofthis aspect include corresponding computer systems, apparatus, andcomputer programs recorded on one or more computer storage devices, eachconfigured to perform the actions of the methods.

Implementations may further include one or more of the followingfeatures. The system further including: a controller communicativelyconnected to the camera device, where the controller is configured tomodify an exposure setting of the camera device; and where the processoris further programmed to: instruct the controller to adjust the varioussetting of the camera device on a periodic basis such that the secondimage is captured with a different camera setting than the first image.The various setting may include, but are not limited to, exposuresetting, shutter speed, zoom setting, and other capture settings. Thecontroller may adjust the settings of the camera device on a periodicbasis, a regular basis, and/or based on other criteria, for example,based on the relative positions of the target vehicle at the firstposition and the second position.

In addition, implementations may further include one or more of thefollowing features. The system where the processor includes anapplication-specific integrated circuit (ASIC) processor, and the cameradevice is communicatively coupled to the processor by a wired connectionand/or a wireless connection. Or an implementation where the cameradevice is physically apart from the processor and is communicativelycoupled to the processor with one of a wired and wireless connection.The system where the camera device further operates as an enclosure forthe processor and the memory arranged therein.

Moreover, implementations may further include a system where the cameradevice omits any infrared illumination component. The system furtherincluding a location tracking device configured to stamp the first imagewith a first location of the police vehicle at the first time when thefirst image is captured by the camera device. The system furtherincluding a clock configured to timestamp the first image upon captureby the camera device. The system where the camera device attached to thepolice vehicle includes a plurality of cameras arranged at differentlocations of the police vehicle and configured to operate in acoordinated manner to capture the first image, and where at least one ofthe plurality of cameras includes an unmanned aerial vehicle equippedwith video capture capabilities. The system including: a wirelesscircuitry configured to receive a command from an external system, wherethe command causes the license plate recognition system to capture theimage data, where the external system includes at least one of a remotecommand center, another police vehicle, and a body-camera device.Implementations of the described techniques may include hardware, amethod or process, or computer software on a computer-accessible medium.

One general aspect includes a method for recognizing a license plate ofa target vehicle, the method including: receive, by a processor locatedat a police vehicle, a first image of a license plate of the targetvehicle at a first time, where the target vehicle is at a firstposition; receive, by the processor, a second image of the license plateof the target vehicle at a second time, where the target vehicle is at asecond portion that is different from the first position; align thelicense plate in the first image and the license plate in the secondimage; transform the first image and the second image to geometricallyrectify the license plate to accommodate for relative positions of thetarget vehicle to the police vehicle; and execute a temporal noisefilter on the first image and the second image to generate aconsolidated image, where the consolidated image has a higherprobability that characters of the license plate are recognized by acomputerized optical character recognition platform than the licenseplate in the first image. Other embodiments of this aspect includecorresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. Themethod including: detect, by a server communicatively coupled to theprocessor, a first boundary of the license plate in the first image;crop, by the server, the first image to discard outside of the firstboundary of the first image; detect, by the server, a second boundary ofthe license plate in the second image; and crop, by the server, thesecond image to discard outside of the second boundary of the secondimage, where the server includes a chipset that uses artificialintelligence for detect operations. Implementations of the describedtechniques may include hardware, a method or process, or computersoftware on a computer-accessible medium.

One general aspect includes a tangible, non-transitory computer-readablemedium storing executable instructions that, when executed by aprocessor of a license plate recognition system, cause the license platerecognition system to: receive, by the processor, a first image of alicense plate of a target vehicle at a first time, where the targetvehicle is at a first position when the first image is captured by acamera device; receive, by the processor, a second image of the licenseplate of the target vehicle at a second time, where the target vehicleis at a second portion that is different from the first position;detect, by the processor, a first boundary of the license plate in thefirst image; detect, by the processor, a second boundary of the licenseplate in the second image; align the license plate in the first imageand the license plate in the second image; transform the first image andthe second image to geometrically rectify the license plate toaccommodate for relative positions of the target vehicle to the cameradevice; and execute a temporal noise filter on the first image and thesecond image to generate a consolidated image, where the consolidatedimage has a higher probability that characters of the license plate arerecognized by a computerized optical character recognition platform thanthe license plate in the first image. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

The methods and systems of the above-referenced embodiments may alsoinclude other additional elements, steps, computer-executableinstructions or computer-readable data structures. In this regard, otherembodiments are disclosed and claimed herein as well. The details ofthese and other embodiments of the present invention are set forth inthe accompanying drawings and the description below. Other features andadvantages of the invention will be apparent from the description,drawings, and claims.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. In oneexample, a light emitting apparatus mountable to a transportationvehicle is disclosed where the apparatus includes a body and a camera.The light emitting apparatus also includes a light source including aplurality of light emitting diodes configured to emit light. The lightemitting apparatus also includes a micro-controller communicativelycoupled to the light source and the camera, where the micro-controlleris configured to dynamically adjust at least illumination power of thelight source and exposure time of the camera.

Implementations may include one or more of the following features. Theapparatus where the light source is configured to emit light in aninfrared frequency range. The apparatus where the dynamically adjustingof the illumination power of the light source and the exposure time ofthe camera occurs at a pre-defined interval of time. The apparatus wherethe dynamically adjusting of the illumination power of the light sourceand the exposure time of the camera occurs repeatedly through a range ofcombinations of illumination power and exposure times. The apparatuswhere the dynamically adjusting of the illumination power of the lightsource and the exposure time of the camera occurs repeatedly through arange of combinations of illumination power and exposure times withoutcommunicating with at least one of a distance measurement component anda speed delta measurement component. The apparatus where the distancemeasurement component and the speed measurement component include aprocessor and a memory that store a plurality of images captured withthe camera, where the speed measurement component compares the pluralityof captured images to determine a distance change over a period of time.Implementations of the described techniques may include hardware, amethod or process, or computer software on a computer-accessible medium.

One general aspect includes a method for generating an enhancedillumination pattern from a light emitting apparatus mounted to asubject vehicle while it is traveling, the method including: measuring,by the light emitting apparatus, an approximate distance to a targetvehicle in a lane near one on which the subject vehicle is traveling.The method also includes calculating, by the light emitting apparatus, arelative speed of the target vehicle in the lane relative to a speed ofthe subject vehicle in its own lane. The method also includes inputtingthe approximate distance to and the relative speed of the target vehicleinto a micro-controller in the light emitting apparatus. The method alsoincludes based on the received inputs, adjusting, by themicro-controller, one or more settings of a camera communicativelycoupled to the light emitting apparatus. The method also includes basedon the received inputs, sending, by the micro-controller, anillumination command to a light source in the light emitting apparatuscorresponding to one of a low, medium, or high illumination. The methodalso includes sending a plurality of images captured by the camera whileoperating with different settings and under different illuminations to aprocessor for selection of an optimal image. Other embodiments of thisaspect include corresponding computer systems, apparatus, and computerprograms recorded on one or more computer storage devices, eachconfigured to perform the actions of the methods.

Implementations may include one or more of the following features. Themethod where the one or more settings of the camera include shutterspeed, ISO, auto-focus, and ultraviolet filter. The method where thelight emitted by the light source is in an infrared frequency range. Themethod where the generating by micro-controller of the illuminationcommand includes: outputting a medium value for the illumination commandwhen the relative speed is below a threshold speed and the approximatedistance is above a threshold distance. The method may also includeoutputting a medium value for the illumination command when the relativespeed is above a threshold speed and the approximate distance is below athreshold distance. The method may also include outputting a high valuefor the illumination command when the relative speed is above athreshold speed and the approximate distance is above a thresholddistance. The method may also include outputting a low value for theillumination command when the relative speed is below a threshold speedand the approximate distance is below a threshold distance.Implementations of the described techniques may include hardware, amethod or process, or computer software on a computer-accessible medium.

One general aspect includes a light emitting apparatus mountable to atransportation vehicle, the apparatus including: a body. The lightemitting apparatus also includes a camera. The light emitting apparatusalso includes a light source including a plurality of light emittingdiodes (LEDs) oriented in a grid pattern inside the body and configuredto emit light in an infrared frequency range. The light emittingapparatus also includes a micro-controller communicatively coupled tothe light source and the camera, where the micro-controller isconfigured to dynamically adjust at least illumination power of thelight source and exposure time of the camera through a rotating list ofcombinations of illumination power and exposure times. The lightemitting apparatus also includes where the light emitted by theplurality of LEDs creates an illumination pattern towards a lane near tothe one on which the vehicle travels. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

The methods and systems of the above-referenced embodiments may alsoinclude other additional elements, steps, computer-executableinstructions or computer-readable data structures. In this regard, otherembodiments are disclosed and claimed herein as well. The details ofthese and other embodiments of the present invention are set forth inthe accompanying drawings and the description below. Other features andadvantages of the invention will be apparent from the description,drawings, and claims.

BRIEF DESCRIPTION OF FIGURES

The present invention is illustrated by way of example, and is notlimited by, the accompanying figures in which like reference numeralsindicate similar elements and in which:

FIG. 1A and FIG. 1B illustrate an illustrative roadway on which subjectvehicles and target vehicles may be traveling while the subject vehicleis operating in accordance with one or more example embodiments.

FIG. 2A shows illustrative camera apparatuses with several technicalcomponents in accordance with one or more examples embodiments.

FIG. 2B is an example of implementations of the camera assembly inaccordance with one or more example embodiments.

FIG. 2C is an illustrative light emitting apparatus in accordance withone or more example embodiments.

FIG. 2D shows illustrative camera apparatuses with several technicalcomponents in accordance with one or more examples embodiments.

FIG. 2E is an illustrative light emitting apparatus in accordance withone or more example embodiments.

FIG. 2F is an illustrative light emitting apparatus in accordance withone or more example embodiments.

FIG. 3 is an example of implementations of the camera assembly inaccordance with one or more example embodiments.

FIG. 4 illustrates an example of a networked environment forimplementation of one or more aspects of the disclosure in accordancewith one or more example embodiments.

FIG. 5 illustrates an example of artificial intelligence methods thatmay be implemented in accordance with one or more example embodiments.

FIG. 6A is a photograph of a street with following traffic and oncomingtraffic in accordance with one or more example embodiments.

FIG. 6B is a long-exposure image and short-exposure image of a field ofview of a street with following traffic and oncoming traffic inaccordance with one or more example embodiments.

FIG. 6C is a subsequent long-exposure image and short-exposure image ofa field of view of a street with following traffic and oncoming trafficin accordance with one or more example embodiments.

FIG. 7 shows photographs of license plates as originally captured andafter aligned and filtered in accordance with one or more exampleembodiments.

FIG. 8 is a comparison of a photograph of an unfiltered image and afiltered image in accordance with one or more example embodiments.

FIG. 9A, FIG. 9B, and FIG. 9C show photographs of license plates asoriginally captured and after aligned and filtered across multipleframes in accordance with one or more example embodiments.

FIG. 10 illustrates edge detection in accordance with one or moreexample embodiments.

FIG. 11 depicts serial and parallel configurations for image capturewith an image sensor in accordance with one or more example embodiments.

FIG. 12 is a flowchart in accordance with one or more examples ofmulti-exposure capture with an LPR system.

FIG. 13 is a flowchart in accordance with one or more examples ofmerging of multiple frames from an image stream to enhance an LPRsystem.

FIG. 14 is a flowchart in accordance with one or more exampleembodiments.

FIG. 15 depicts illustrative operational settings based on arelationship between speed delta and distance in accordance with one ormore examples embodiments.

DETAILED DESCRIPTION

Generally, systems and methods are disclosed for capturing the licenseplate information of a vehicle in relative motion to a camera device. Inone example, the camera device captures an image of the vehicle'slicense plate across multiple frames. The camera system detects thelicense plate in the multiple frames, then aligns and geometricallyrectifies the image of the license plate by scaling, warping, rotating,and/or performing other functions on the images of the license plate.The camera system optimizes the capturing of the license plateinformation by executing a temporal noise filter (e.g., temporal noisereduction—TNR) on the aligned, geometrically rectified images togenerate a composite image of the license plate for optical characterrecognition. In some examples, the camera device may include a highdynamic range (HDR) sensor that has been modified to set the longexposure and short exposure of the HDR sensor to capture an image of avehicle's license plate, but without the HDR sensor consolidating theimages into a composite image. The camera system may set optimalexposure settings based on detected relative speed of the vehicle orother criteria.

By way of example, and in no way limiting the features and contemplatedcombination of features disclosed herein, four illustrative use casesare described below describing particular aspects of disclosed features.In addition to the four use cases listed below, the disclosurecontemplates many other examples, embodiments, implementations, and usecases that use combinations of the features and aspects described in theindividual use cases. For example, one or more use cases describe acamera device positioned in/on the camera car and that iscommunicatively coupled to a processor in the automatic license placereading (ALPR) system by a wired connection and/or a wirelessconnection. The terms ALPR and LPR are used interchangeably in thisdisclosure. The use cases may also operate in an environment where thecamera device is physically apart from the processor and iscommunicatively coupled to the processor with one of a wired andwireless connection. For example, in one example the camera deviceattached to the police vehicle includes a plurality of cameras arrangedat different locations of the police vehicle and configured to operatein a coordinated manner to capture images of vehicle license plates orother items. Moreover, in some examples, at least one of theaforementioned plurality of cameras may include an unmanned aerialvehicle (UAV) equipped with video capture capabilities. The UAV may bemounted to the vehicle and may be automatically launched as appropriateby the LPR system upon occurrence of particular trigger events.

In addition, one or more embodiments include computerized methods,systems, devices, and apparatuses that capture images of one or moremoving vehicles (i.e., a target vehicle) from another moving vehicle(i.e., subject vehicle). The disclosed system dynamically adjustsillumination power, exposure times, and/or other settings to optimizeimage capture that takes into account distance and speed. By optimizingfor distances and moving vehicles, the disclosed system improves theprobability of capturing a legible, usable photographic image. In oneexample, the disclosed system may be incorporated into an asymmetriclicense plate reading (ALPR) system. Aspects of the disclosed systemimprove over the art because, inter alia, it dynamically adjustsillumination power, exposure times, and/or other settings to optimizeimage capture that takes into account distance and speed. In oneexample, the disclosed system may be incorporated into an asymmetriclicense plate reading (ALPR) system. For example, by optimizing fordistances and moving vehicles—the disclosed system improves theprobability of capturing a legible, usable photographic image of atarget vehicle's license plate (or other information such as an image ofa driver and/or passengers in a vehicle). Moreover, aspects of thedisclosed system improve the camera's ability to capture objects andlicense plates at farther distances (e.g., more than 20-30 feet away)than existing technology.

Regarding FIG. 1A, in practice, target vehicles (e.g., oncoming traffic)on a roadway 102 traveling in a direction opposite to a subject vehicleon the roadway 104 may be traveling at different speeds and be atdifferent distances. Meanwhile, the continuous flow of new targetvehicles on the roadway 102 (e.g., oncoming traffic and followingtraffic) adds complexity to image capture. The optimum value for camerasettings for each scenario is a non-linear function which depends on thecamera performance parameters and detection algorithms, and may provideimages of sufficient quality to capture objects and license platenumbers. In scenarios such as FIG. 1 , which corresponds to roadways incountries like Britain and India, the oncoming traffic is on theright-hand side lane; thus, the configuration set for the implementationwould be reversed for those countries.

FIG. 2D illustrates a camera apparatus enhanced with various aspects ofthe disclosed systems. The camera apparatus 201 may include one or morecomponents to assist in enhancing image capture of a license plate of amoving, target vehicle. In particular, a micro-controller 204 may beincorporated with the camera apparatus 201 to automatically adjustsettings. The micro-controller 204 may adjust settings such as, but notlimited to, exposure time, (optionally) illumination power, focusposition, sensor gain (camera ISO speed), aperture size, filters (e.g.,ultraviolet—UV), image-noise filtering, and the like.

Elaborating upon the examples provided with the aid of FIG. 15 withrespect to asymmetric illumination to enhance license place recognition,the micro-controller 204 may receive inputs of speed delta and distance,and adjust the settings of exposure time and/or illumination poweraccording to various scenarios 1500 identified. For example, in scenario(A) in the lower right-corner of the graph 1500, the target vehicleswith small speed delta (compared to the subject vehicle) but withmaximum recognition distance, cause the micro-controller 204 to set thecamera to use the longest exposure times and/or medium illuminationpower. The longer exposure time is optimal because the angular movementis minimized due to long distance and small speed delta. Due to thelong-exposure time, the illumination power does not need to be thehighest possible, even when the target vehicle is at far distance.Example values are 4 millisecond and 0.5 W illumination power,respectively.

With reference to FIG. 15 , in another example, in scenario B in theupper right-corner of the graph 1500, the target vehicle is at a longdistance with large speed delta need medium exposure time because thespeed delta pushes it shorter, but long distance pushes it longer. Thesetarget vehicles need the highest illumination power available tocompensate for the shorter exposure time compared to scenario A. Themicro-controller 204 may also increase gain, in some examples, than inscenario A if the illumination power reserve is running out. Examplevalues are 2 millisecond and 1 W illumination power, respectively.

With reference to FIG. 15 , in yet another example, in scenario C in theupper left-corner of the graph 1500, the target vehicle is at a shortdistance and a high speed delta creates the highest angular speed in thecamera view. Therefore, the micro-controller 204 sets the exposure timeto be very short, (e.g., only 0.1 milliseconds). As a result, theshutter covering the image sensor 202 may open for only a very shorttime. As the target vehicles are close in distance and the power of theillumination is proportional to the inverse of the distance squared, theillumination power can be in the medium level, such as 0.25 Willumination power.

With reference to FIG. 15 , in another example, in scenario D in thelower left-corner of the graph 1500, the target vehicle is at a shortdistance with a small speed delta. Thus, the micro-controller 204 sets amedium exposure time, similar to its operation in scenario B. Theillumination power can also be set to a minimum due to the shortdistance (similar to scenario C), but longer exposure time, e.g., 0.05W. Coincidentally, static LED illumination cone optimization (see FIG.1B) supports this behaviour—vehicles expected to need lower illumination(e.g., scenarios A and D) have a lower power illumination cone.

Referring to FIG. 2D, in some examples, the camera apparatus 201 may beintegrated with a light source 220 for emitting infrared light, or lightin a different frequency spectrum. In alternate embodiments, a lightemitting apparatus 230 may be physically separate from the cameraapparatus 201. In such embodiments, the micro-controller 204 in thecamera apparatus communicates with the micro-controller in the lightemitting apparatus 230. For example, if the camera apparatus 201 isequipped with components to detect and measure the delta speed value anddistance value of a target vehicle, then its micro-controller 204 mayshare this information with the light emitting apparatus 230 forefficiency. The apparatuses may share information wirelessly usingantenna and wireless circuitry 208. The wireless circuitry 208 maysupport high-speed short-range communication to permit fastcommunication between the apparatuses. The wireless circuitry 208 mayalso include long-range communication hardware to permit connection to aremote server computer or cloud devices.

In addition to efficiency, the sharing of information between thedevices furthers the synchronization of the apparatuses 201, 230 forpurposes of capturing a higher quality image. For example, if the cameraapparatus 201 relies on the light emitting apparatus 230 to provide apulse of infrared light at the moment of, or just immediately prior to,the shutter 203 on the camera apparatus 201 opening, the two apparatusmust communication and synchronize. In one example, to aid insynchronization, inter alia, the camera assembly may operate apre-defined sequence of configuration settings at pre-defined intervals.The system may cycle through a set of scenarios (e.g., scenarios A-D inFIG. 15 ) to test the quality of image capture with each scenario.Meanwhile, multiple settings may be used without requiring the separateapparatus to synchronize each time—rather, the separate apparatus mightsynchronize just at the start of the pre-defined script. Once the scriptbegins execution, each apparatus performs its part to completion.

Light source 220 (or light emitting apparatus 230) providesfunctionality to the overall system because it provides the illuminationpattern for improving image capture quality. As such, thesynchronization or alignment of the light emitting apparatus 230 and thecamera apparatus 201 is important. In one example, an LED pulse andcamera exposure time are aligned to capture numerous images with varyingconfiguration settings. For example, first, the micro-controller 204uses the most powerful LED pulse available and longer exposure time.This is good for catching target vehicles at longer distances (because alot of light is needed and also the angular velocity is smaller so thelonger exposure time is acceptable). Then on the next frame, themicro-controller 204 uses medium exposure time and illumination pulsepower. This is useful for catching target vehicles at medium distances.Next, the micro-controller 204 may set a very short exposure time andalso the lowest power LED pulse to catch the closest vehicles. Then thecycle may start again with the longest exposure time and highest pulsepower. By adjusting both the exposure time and pulse power, the systemis optimized for “inversely proportional to the square of the distance”characteristics of these systems. The illumination intensity isinversely proportional to the square of distance between the lightsource and target vehicle's license plate. This makes the exposure verydifficult—if the target car is slightly too far away, the license platemay be too dark to see, and if the car is slightly too close, thelicense plate may be too bright to see (i.e., overexposed).

Referring to FIG. 2D, the camera apparatus 201 may also include memory210, a global positioning system (GPS) unit 212, and a processor 214.The memory 210 is a suitable device configured to store data for accessby a processor, controller, or other computer component. A memory storesinformation. A memory may provide previously stored informationresponsive to a request for information. A memory may store informationin any conventional format. A memory may store electronic digitalinformation. A memory may provide stored data as digital information. Amemory includes any semiconductor, magnetic, optical technology, orcombination thereof for storing information. A memory may receiveinformation from a processing circuit for storage. A processing circuitmay provide a memory a request for previously stored information.Responsive to the request the memory may provide stored information to aprocessing circuit. A memory may include any circuitry for storingprogram instructions and/or data. Storage may be organized in anyconventional manner (e.g., program code, buffer, circular buffer).Memory may be incorporated in and/or accessible by a transmitter, areceiver, a transceiver, a sensor, a controller, and a processingcircuit (e.g., processors, sequential logic). A memory may perform thefunctions of a data store and/or a computer-readable medium. The memory210 may include non-volatile memory (e.g., flash memory), volatilememory (e.g. random-access memory—RAM), or a hybrid form ofcomputer-readable medium for data storage. Moreover, the memory 210 mayinclude one or more cache memories for high-speed access.

In various embodiments, processor 214 may comprise any circuitry,electrical components, electronic components, software, and/or the likeconfigured to perform various operations and functions discussed herein.For example, processor 214 may comprise a processing circuit, aprocessor, a digital signal processor, a microcontroller, amicroprocessor, an application-specific integrated circuit (ASIC), aprogrammable logic device, logic circuitry, state machines,micro-electromechanical system (MEMS) devices, signal conditioningcircuitry, communication circuitry, a computer, a computer-based system,a radio, a network appliance, a data bus, an address bus, and/or anycombination thereof. In various embodiments, processor 214 may includepassive electronic devices (e.g., resistors, capacitors, inductors,etc.) and/or active electronic devices (e.g., op amps, comparators,analog-to-digital converters, digital-to-analog converters, programmablelogic, sample rate converters (SRCs), transistors, etc.). In variousembodiments, processor 214 may include data buses, output ports, inputports, timers, memory, arithmetic units, and/or the like.

In rapid operation, a camera apparatus 201 may capture multiple imagesin a matter of seconds. Multiple levels of cache memory may be used toensure efficient execution. The memory 210 may closely operate with theprocessor 214. For example, the processor may include an image processorto analyze images captured by the apparatus 201 to determine if theimage is sufficiently legible or insufficiently legible. The imageprocessor may analyze images to determine whether to retain the imagedata, or immediately discard the image data. At least one benefit of animage processor operating nearly simultaneously with image capture isreduced memory usage due to immediate discarding of useless or emptyimages.

In one example of technological efficiencies of the system, the imagecaptured by the image sensor 202 may be stored in memory 210 and thensent to processor 214 to detect the vehicle license plate number of thetarget vehicle in the image. The vehicle license plate number may thenbe compared against a database of license plate numbers (or otherinformation) associated with possible legal-related issues. In someembodiments, the vehicle license plate number (and other information)may be sent over a network to a remote server in the cloud that stores adatabase of license plate numbers. If a concern is identified, theoperator of the subject vehicle may be alerted audibly, visually, orthrough haptic feedback (e.g., vibrations).

In addition, the camera apparatus 201 may include a GPS unit 212 tocapture the location of the camera apparatus 201 at the instant an imageis captured. In addition to location, the GPS unit or other component inthe camera apparatus may timestamp the capture of the image. Locationand time data may then be embedded, or otherwise securely integrated,into the image (e.g., metadata of the image) to authenticate the captureof the photograph. Once the image is securely stamped with location anddate/time, the image may, in some example, be securely transmitted to acloud server for storage. In some examples, the image may be stored inan evidence management system provided as a cloud-based service.

In addition to location-stamping the image, the GPS unit 212 may also beused to enhance image capture. In one example, the speed of the subjectvehicle may be obtained from the GPS unit 212 or from the on-boarddiagnostics (OBD) port of the subject vehicle. The vehicle speed and/orthe positional data (e.g., longitude-latitude data) from the GPS unit212, may allow the micro-controller to predict whether the subjectvehicle is on a rural highway or other street. The speed of the subjectvehicle effects the quality of the images captured because the angularvelocity for close target vehicles will be too high. Therefore, thesystem becomes trained about which settings are optimal for thescenario. For example, the GPS unit 212 may detect if the subjectvehicle is traveling in a city, suburb, or rural area, and adjust thesettings in adherence.

In addition to location-stamping the image, the GPS unit 212 may also beused to enhance image capture. In one example, the system may rememberparticular configuration settings at a particular geographic location,and the micro-controller 304 may re-use the prior ideal configurationsettings at that location. For example, a particular stretch of highwaymight have an impenetrable row of trees that renders the system futilefor a duration of time. During that time, the system may halt imagecapture if the system is primarily being used in an ALPR application.Rather than collect image data and consume limited memory 210 on thecamera apparatus 201, the system uses historical data to learn andimprove the operation of the system with a feedback loop.

Referring to FIG. 2A, the camera apparatus 201 may include and/or omitone or more components in some embodiments. For example, the lightsource 220 may be omitted in some embodiments of the camera apparatus201. Instead, the light emitting apparatus may be external to the cameraapparatus 201 and operate in a synchronized manner with the cameraapparatus 201. Furthermore, the camera apparatus 201 may includeadditional components 218, such as a stabilizer, optical zoom hardware,cache memory, interface to a vehicle's on-board diagnostics (OBD) port,multi-axis accelerometer, a motion sensor, and components 216 configuredto use artificial intelligence (AI) to perform operations. For example,an AI model may be trained and stored in memory on the camera apparatus201 to assist the AI component 216 to use a feedback loop to adjust andrefine its settings and operation. The AI component 216 may include agraphics processing unit (GPU) for processing machine learning and deeplearning calculations with efficiency and speed. As illustrated in FIG.5 , a neural network 500 executing in the GPU can provide valuablefeedback as the system is trained with real image captures.

Furthermore, in a networked, crowdsourced arrangement, the cameraassembly system may be installed on multiple, subject vehicles operatingin a particular geographic area to provide broader coverage. Theplurality of camera apparatuses on different vehicles may cooperate witheach other by sharing information over a wireless connection. The cameraapparatus in a first subject vehicle may be operated in conjunction withglobal satellites or other location tracking systems. A second subjectvehicle with a camera assembly system may share information eitherdirectly with, or via a cloud server, the first subject vehicle. Thesharing of information may allow the training of the AI component 216with greater efficiency.

Although several of the examples with reference to FIGS. 2A and 2D havementioned illumination with a light source 220, not all implementationsof the camera apparatus (i.e., camera device) need to include such acomponent. For example, with respect to an LPR system that capturesvehicle license plates using multi-exposure capture and/or temporalnoise filtering (TNF), one or more of the components in the cameradevice 201 may be present, but not necessarily all components. Forexample, an LPR system may efficiently capture images in low lightingconditions without a light source 220 or light apparatus 230.

Regarding FIG. 4 , in one or more arrangements, teachings of the presentdisclosure may be implemented with system of networked computingdevices. FIG. 4 illustrates that the camera assembly 201 may operatewith other networked computing devices 412, 414. In addition to thedevice shown in FIG. 4 , other accessories and devices may becommunicatively coupled to the camera assembly 201. For example, anoperator (as illustrated in FIG. 3 ), such as a law enforcement officer,may be associated with one or more devices 322. The devices may include,but are not limited to, a wearable camera, a weapon 322, and variousdevices associated with a vehicle 108, such as a vehicle-mounted camera201. The weapon 322 may be, for example, a conducted energy weapon (CEW)that transmits notifications regarding events such as firing events,cartridge loading, holster removal, and/or the like. Other devices, suchas a heart rate sensor device, a holster sensor device, and/or the likemay also be included in the system but are not illustrated in FIG. 4 .

The system includes an evidence management system 414 having a digitalvideo and audio processing system with an audio watermark processingengine, such as the digital video and audio processing system. Thedigital video and audio processing system may be configured to receiveand process audio watermarks, and may also include a synchronizationengine. Some of the devices in FIG. 4 may have limited communicationfunctionality. For example, devices may have short-range wirelesscommunication abilities, but some devices may only be able to perform adirect long-range transmission or reception of information, such as toan evidence management system 414, when physically connected to anevidence collection dock that communicates with the evidence managementsystem 414 via a network such as a local area network (LAN), a wide areanetwork (WAN), and/or the Internet.

In some embodiments, a computing device 412 is provided at the vehicle108. The computing device 412 may be a laptop computing device, a tabletcomputing device, or any other suitable computing device capable ofperforming actions described herein. The computing device 412 may becapable of short-range communication with the devices in the system, andmay also be capable of long range communication with the evidencemanagement system 414, a dispatch system, or any other system. In someembodiments, the computing device 412 has the components andcapabilities described herein.

Communication between devices 201, 412, 414 may include any conventionaltechnologies (e.g., cellular phone service, text and data messaging,email, voice over IP, push-to-talk, video over cellular, video over IP,and/or the like). Communication may use conventional public or privatemedia (e.g., public cellular phone service, local area service, reservedchannels, private trunk service, emergency services radio bands, and/orthe like). In some embodiments, the device 412 may be configured toperform computationally intensive operations as an edge computingdevice, thus reducing the load on and bandwidth to remote device 414.

Computing device 412 may be located in or around a subject vehicle. Thecomputing device 412 may communicate with an on-board diagnostics (OBD)port of the subject vehicle to collect information about speed and otherproperties of the subject vehicle. In some examples, the device 412 maycommunicate wirelessly with vehicle sensors positioned in the subjectvehicle. The data collected about the subject vehicle may be stored inassociation with images captured by the camera assembly 201.

Computing device 412 may include a GPU for performing machine learning(ML) computations using training data 416 collected by the cameraassembly 201 and other camera assemblies mounted on other vehicles.Through the collection of this data, the neural network 500 illustratedin FIG. 5 provides feedback to the system to improve performance.

FIG. 5 illustrates a neural network 500 that may be executed by module410 in device 414, including providing other artificial intelligencecomputations. At least one advantage of the module 410 being located inthe cloud is that edge-computing resources may be conserved for othercomputations. Examples of edge-computing resources in the system includecomponent 216 in the camera apparatus 201. For example, an AI model maybe trained and stored in memory on the camera apparatus 201 to assistthe AI component 216 to use a feedback loop to adjust and refine itssettings and operation. The AI component 216 may include a GPU forprocessing machine learning and deep learning calculations withefficiency and speed. For example, the deep learning systems 500 mayanalyze and categorize video based on its fundamental sensorycomponents: what's visually present (sight) and what's happening acrosstime (motion). Examples of motion include the way objects move acrosstime to derive deeper meaning from the scene. For example, the deeplearning can determine if an object is stationary or moving, whatdirection it's moving, and how the scene evolves around it.

FIG. 5 may further include a prediction subsystem for using model datato train the neural network 500 to predict whether an image will beoptimal with particular camera parameter settings and illumination conepattern settings. It should be noted that, while one or more operationsare described herein as being performed by particular components, thoseoperations may, in some embodiments, be performed by other components orby other devices in the system. In addition, although some embodimentsare described herein with respect to machine learning models, otherprediction models (e.g., statistical models or other analytics models)may be used in lieu of or in addition to machine learning models inother embodiments (e.g., a statistical model replacing a machinelearning model and a non-statistical model replacing anon-machine-learning model in one or more embodiments). In someembodiments, techniques used by the machine learning models (or otherprediction models) include clustering, principal component analysis,nearest neighbors, and other techniques. Training of machine learningmodels (or other prediction models) may include supervised orunsupervised training.

In some embodiments, a neural network may be trained and utilized forpredicting optimal setting configurations. As an example, neuralnetworks may be based on a large collection of neural units (orartificial neurons). In some embodiments, each individual neural unitmay have a summation function which combines the values of all itsinputs together. In some embodiments, each connection (or the neuralunit itself) may have a threshold function such that the signal mustsurpass the threshold before it is allowed to propagate to other neuralunits. These neural network systems may be self-learning and trained,rather than explicitly programmed, and can perform significantly betterin certain areas of problem solving, as compared to traditional computerprograms. In some embodiments, neural networks may include multiplelayers (e.g., where a signal path traverses from front layers to backlayers). In some embodiments, back propagation techniques may beutilized by the neural networks, where forward stimulation is used toreset weights on the “Layer 1” neural units. In some embodiments,stimulation and inhibition for neural networks may be more free-flowing,with connections interacting in a more chaotic and complex fashion.

Referring to FIG. 3 , in some embodiments, the camera apparatus is amountable camera that provides a point of view associated with thesubject vehicle. In some embodiments, the camera apparatus may bemodified to be a device carried by the user, such as mounted onto ahelmet. In one example (see FIG. 3 ), the camera assembly 201 mayautomatically start, pause, stop, etc. based on events received via ashort-range wireless interface with the vehicle sensors of vehicle 108.For example, if a subject vehicle 108 is standing still at rest, thevehicle's speed delta may register at a lower value. As such, referringto the scenarios 1500 in FIG. 15 , the camera apparatus 201 may adjustits settings configuration via the micro-controller to accommodate theenvironment. Meanwhile, the memory 210 store may also store eventinformation in association with captured images to record conditions atthe time of image capture. This information may be useful when auditingdata for potential use in a legal proceeding.

Moreover, connecting with a local network may provide the device 201with event notifications, such as when the operator opens the car door,activates a police car's light/siren bar, and other events, so thedevice 201 can react accordingly. For example, the LPR system mayautomatically turn ON or OFF the camera device based on the lawenforcement vehicle's status—e.g., if siren alarm is ON, if siren lightsare ON, if the vehicle is driving at high speeds, whenever movement isdetected. In addition, one or more features disclosed herein may be, insome appropriate examples, embodied in a bodycam worn on a policeofficer. In such embodiments, the functionality may be purposefullyculled to accommodate a smaller battery. It may also be embodied in adrone (UAV) or other mobile device. An external system may send acommand to the processor of the LPR system to cause the processor toactivate and capture the first image, then the second and subsequentimages. In some examples, the external system may comprise at least oneof a remote command center, another police vehicle, and/or a body-cameradevice. Meanwhile, when multiple vehicle license plates are detected ina single image capture, the LPR system might attempt to simultaneouslyperform the operations for each of the plates.

Regarding the subject vehicle, it may be a police patrol car, but can beany road or off-road vehicle (or even flying vehicle), including jeeps,trucks, motorcycles, ambulances, buses, recreational vehicles, fireengines, drones, and the like. The target one or more vehicles canlikewise be any combination of any types of vehicles, and will be in theproximity of the subject vehicle in any of numerous differentplacements. Some of the target vehicles will have rear license plates,front license plates, or both front and rear plates.

Regarding mounting locations, one or more cameras may be mounted at thefront and/or rear portions of the subject vehicle. Mounting can be onthe bumpers or anywhere else, and can even be located in other positionssuch as in the siren tower on top of the subject vehicle or inside thecab behind the windshield. The one or more cameras can be mounted in thecenter line of the subject vehicle, or off-center in any suitablemanner. The one or more cameras may be mounted above the AmericanStandard 1 (AS1) line of a windshield so as to not obscure a vehicleoperator's view through the windshield. The at least one camera providesfront, rear, side, and/or a combination of coverage. A second, third, ormore other cameras may optionally be included on the subject vehicle. Insome embodiments, a plurality of cameras may be mounted on the subjectvehicle in suitable locations (e.g., front, rear, side, or top) to allowup to 360 degrees of field of view for image capture. Moreover, thecamera assembly may be programmed to operate autonomously in backgroundmode, e.g., without requiring operator input. The camera assembly may,in some embodiments, only alert the operator when the camera assemblyhas identified a possible safety (or legal-related) concern, forexample, using the captured license plate information of neighboringvehicles. The camera assembly may, in some embodiments, operatecontinuously for an extended period of time while the subject vehicle ispatrolling an area, and can be turned on and off by the operator asdesired.

Referring to FIG. 12 , the first of several use cases describe oneillustrative image sensor (e.g., an HDR sensor) being used in arelatively static scene that operates using HDR (and/or HDR-like)features of a camera device of an LPR system that is installed in or ona vehicle (or other non-stationary/mobile device). In some embodiments,the image sensor may be installed at a fixed location, such as a lightpost, building exterior, etc. In one example, the image streamsoutputted by the HDR sensor are modified to keep the outputted imagestreams separate. An HDR sensor can generate two or more video streamswith different exposure times from a single image sensor. This can bedone by the sensor HDR mode. For example, a first set of images may becaptured by the camera device using its image sensor, and that first setof images may comprise a first long-exposure time image and a firstshort exposure time image. The LPR system then receives (in step 1202)the long-exposure image and also receives (in step 1204) theshort-exposure image. In normal HDR mode, however, the long and shortexposures are combined. In this example, however, the LPR systemmaintains separate streams. The long and short streams use differentexposure times for different frames in the video stream. In other words,the image sensor captures images, but keeps the long-exposure andshort-exposure image streams separate—i.e., the frames are notconsolidated the typical way that HDR mode operates. There is a greaterlikelihood of accurate OCR of a license plate of a target vehicle withgreater relative speed in the short-exposure image.

In some embodiments, no speed detection (e.g., no relative speed of thelicense plate in the image is determined) or consideration is performedby the LPR system, thus no steps are taken to further optimize theexposure setting(s) of the HDR sensor for each stream. In otherexamples, speed detection or consideration is performed by the LPRsystem, and steps are taken to further optimize the exposure setting(s)of the HDR sensor for each stream. In one embodiment, the long-exposuresetting and short-exposure setting of the image sensor may each beadjusted (e.g., based on detected relative speeds or by a predeterminedamount), and a second, subsequent pair of images are captured. Thesubsequent pair of images may comprise a second long-exposure time imageand a second short-exposure time image. There is a greater likelihood ofaccurate OCR of the target vehicle with greater relative speed in theshort-exposure image. Moreover, when the image sensor's short-exposuresetting is refined/adjusted to account for relative speeds, the accuracyof the OCR may further improve. Below is one illustrative HDR use caseinvolving the technical components and method steps disclosed herein.Although an HDR sensor is mentioned in various examples, the examplesare not so limited—any image sensor with the desired capabilitiesdescribed herein may be substituted for the HDR sensor.

In particular, in low light scenarios, an LPR system may face difficultyin accurately recognizing license plate characters due to insufficientlighting. To improve the exposure, the camera ISO can be increased, orthe shutter speed can be reduced (i.e., exposure time is increased).However, with increased ISO comes increased noise, which can reduce OCRperformance. Additionally, with slower shutter speeds (i.e., longerexposure times), images may become blurred. This problem is exacerbatedwhen the relative speed between the camera-equipped vehicle and thetarget vehicle is high, such as in the case of oncoming traffic.Therefore, in this use case, with a single camera with a single field ofview, the image sensor captures two streams of data—one stream havingexposure settings optimized for low relative speed traffic (e.g., samedirection traffic) and the other stream optimized for high relativespeed traffic (e.g., oncoming traffic), as explained in more detailherein.

In an initial step, the LPR system may use one or more object detectionlibraries to find an object in a captured image that matches thecharacteristics of a license plate. The library takes a captured imageas input and identifies a license plate in the image using objectdetection. In one embodiment, a heat map of likelihood of a licenseplate being in the image may be done. In one example, this likelihood(e.g., probability/confidence score) may be done by an artificialintelligence (AI) model trained from image data.

Next, a processor in the LPR system may use one or more object trackinglibraries to detect what appears to be the same license plate in one ormore subsequently captured images. In one example, the LPR system mayuse a boundary of (e.g., bounding box around) the license plate to trackits position across frames. FIG. 10 illustrates how an edge detectionmodule in the object tracking library may detect and track aparallelogram boundary shape 1004 around a license plate. In step 1206of FIG. 12 , the LPR system detects one or more license plates in thestream of frames with long-exposure and/or short-exposure.

Referring to FIG. 6A, an output is depicted of the object detection andobject tracking libraries, after analysis by the LPR system. Oncomingvehicle 602, following vehicle 604, and other vehicles are detected andidentified in FIG. 6A. Box 601 shows portions of the image with oncomingtraffic, and box 603 and box 605 show portions of the image withfollowing traffic. The number of boxes may correspond with a number oflanes of traffic—both oncoming traffic and following traffic. Of course,FIG. 6A represents a later-generated output of the LPR system once thesteps described below have been performed and an optical characterrecognition (OCR) has been completed of the resulting image data. FIG.6A illustrates the capabilities of the object detection and objecttracking libraries to detect the characteristics/attributes of a licenseplate. The object detection and object tracking libraries may be furthertrained to detect characteristics of a license plate based on its knownposition on the front of a vehicle or the rear of a vehicle. Forexample, characteristics of a license plate may include, but are notlimited to, detection of a rectangular shape located on a vehiclecontaining a horizontal arrangement of alphanumeric characters and/orwith a state name located in pre-defined position within the rectangularshape. After performance of this step in this first use case, the LPRsystem may have just the bounding boxes around suspected license platesin the image—the object tracking and OCRing have not yet occurred.

In the next step, multiple subsequent images are captured. The LPRsystem may cause its camera device to randomly/periodically adjustexposure settings of long exposure and short exposure on the HDR sensorto attempt to capture a sharper/higher quality image. The exposuresettings might be adjusted based on one or more other inputs—forexample, a daylight sensor or a clock mechanism to determine when alow-light condition exists and adjusting exposure settings based onlighting conditions. In some examples, the properties of the capturingcamera device could be adjusted automatically or dynamically—e.g., basedon a series rotation or random cycling through a pre-defined set ofsettings.

In one example involving low-light conditions, an LPR system may furtherinclude a camera device that includes an (optional) infraredillumination component 230, as illustrated in FIG. 2D. However, the LPRsystem described herein may be designed to alternatively operate withoutthe assistance of an infrared illumination component. For example, oneproblem with prior art LPR systems was that in low light conditions,license plate images tended to be very noisy, and it wasdifficult/impossible to accurately detect the characters in an image ofa license plate. Moreover, long exposure times were inadequate to fullysolve this problem because when the vehicles are in motion, the capturedimage risked being blurred. However, with the LPR system disclosedherein, the method steps and arrangement of components disclosed hereinovercome the shortcoming in prior art LPR systems. Nevertheless, thedisclosed LPR system may also operate without detriment in combinationwith an (optional) illumination system.

To achieve higher success with legible license plate capture, the LPRsystem may cause its camera device to adjust exposure settings of longexposure and short exposure on the HDR sensor. The settings of thecamera device in the LPR system may be adjusted using a controllercommunicatively connected to the camera device. The controller may beconfigured to modify a setting of the camera device. The various cameradevice settings may include, but are not limited to, exposure timesetting, shutter speed, zoom setting (optical or non-optical zoom),illumination power, focus position, sensor gain (e.g., camera ISOspeed), aperture size, filters, other capture settings, and the like. Aperson of skill in the art will appreciate after review of the entiretydisclosed herein that one or more of the settings may be interrelated ordependent. For example, an exposure of 1/25 sec at f/11, ISO 100 isequivalent to an exposure of 1/400 sec at f/2.8, ISO 100. In otherwords, because the shutter speed has been reduced by four stops, thismeans less light is being captured by the image sensor in the cameraassembly. As a result, the aperture is increased in size by four stopsto allow more light into the camera assembly to maintain constantexposure. While there are benefits and disadvantages to adjusting thesettings in one way versus another, such knowledge would fall within therealm of a person having skill in the art. For example, a person havingskill in the art would understand that to maximize exposure, a cameraassembly might be set to a large aperture, 6400 ISO, and a slow shutterspeed. Meanwhile, to minimize exposure, a camera assembly would be setto a small aperture, 100 ISO, and a fast shutter speed. Of course, thesharpness of the captured image might be effected by depth of field,aperture, and shutter speed settings. In particular, with mostembodiments disclosed herein involving a moving subject vehicle capturean image of a moving target vehicle, the ability to capture an imagewithout introducing blurriness or shading or planar warp is aconsideration.

The processor in the LPR system may be programmed to instruct theaforementioned controller to adjust the various setting of the cameradevice on a periodic basis, regular basis, random basis, and/or othercriteria such that the second image is captured with a different camerasetting than the first image. In one example, groupings for exposuretime, illumination power, and/or other settings may be simultaneouslyadjusted for different operating scenarios. In another example, thecriteria may be based on the relative positions of the target vehicle ata first position in an image and at a second position in a subsequentlycaptured image. In one example, the relative position is thedelta/change in position of the license plate from the first image to asubsequent, second image and takes into account the position of the LPRsystem and the target vehicle with the license plate affixed thereon ateach of the image capture events. For example, referring to FIG. 6A,relative speeds of vehicles may be detected taking into considerationthat box 601 shows portions of the image with oncoming traffic, and box603 and box 605 show portions of the image with following traffic.Therefore, the speed of vehicles in box 601 relative to the capturingcamera device may be increased, while the relative speed of vehicles inbox 603 and box 605 may not be increased because the direction of motionof the camera device is in the same direction as vehicle 604. In someexamples, the LPR system may use motion blur analysis of the licenseplate in one or more of the captured multi-exposure images to calculatethe aforementioned relative speed. Then, the LPR system may adjust theimage sensor to capture subsequent images with an adjusted exposuresetting, accordingly, to reduce motion blur.

In another example, the LPR system may be pre-programmed to instruct thecontroller to modify one or more capture settings of the camera devicebased on the relative positions of the target vehicle at a firstposition in an image and at a second position in a subsequent image,such that the change in the relative positions of the target vehicle inthe images shows relative speed. For example, the relative speed of atarget vehicle with a license plate is calculated by determining apixels per second change in the license plate across consecutive imagesof the license plate captured by the image sensor. As a result, anotherimage may be subsequently captured at a different capture setting thanthe first image based on the relative positions indicative of relativespeed. The relative speed may be measured, in some examples, in units ofpixels/second on the image rather than traditional speed units of milesper hour (mph) or kilometers per hour (kph). In other words, therelative speed of the vehicle might not be calculated in km/hr, but thespeed of movements of visual features in the pixel space, e.g., deltapixel/second of a fixed point (e.g., top right corner) of the licenseplate. At least one advantage is that the latter is easier to compute.The algorithm for determining exposure setting is interested in speeddelta in pixels/s for blur estimation, and it is more efficient tocalculate than accurately measuring km/h. The speed delta in e.g., km/hof a car can be derived from this information, if lens details (e.g.,field of view, focal length, blur analysis, distortion model) andlicense plate size/dimension and location is known.

Referring to FIG. 6B, with respect to vehicle 606, the system detects afirst license plate in a first portion 603 of the first long-exposuretime image that was captured with a first long-exposure setting. Then,at a later time, the LPR system captures a second long-exposure timeimage, as illustrated in FIG. 6C, and detects the same first licenseplate in a first portion 603 of the second long-exposure time image. Thesecond long-exposure time image may have been captured with the same ordifferent settings as the first long-exposure setting. In some examples,the time between the capture of the images in FIG. 6B and the capture ofthe images in FIG. 6C may be very small such that the LPR system mayperform the steps of the methods disclosed herein in near real-time. Forexample, the processor and controller of the LPR system captures,detects, analyses, and/or adjust the exposure (or other) settings of theimage sensor of the camera device between the capture of images in FIG.6B and FIG. 6C. In some examples, an application specific integratedcircuit (ASIC) processor may be used in the LPR system to improveresponse time.

Next, the LPR system may select a plurality of images for processing andmerging into a consolidated image for reading. As explained in thisdisclosure and with reference to FIG. 11 , in general, there is agreater likelihood of useful OCR of license plates of oncoming traffic(i.e., those with a greater relative speed) in the short-exposure images(as illustrated with step 1210 in FIG. 12 ); meanwhile, there is a lowerlikelihood of useful OCR of license plates of oncoming traffic in thelong-exposure images; and, a lower likelihood of OCR of license platesof oncoming traffic in consolidated images, such as the image resultingfrom the merging of the long-exposure image with the short-exposureimage because such merging may introduce blur. Meanwhile, step 1208 ofFIG. 12 illustrates that when the license plate is on a followingvehicle, then a long-exposure image may provide more useful OCR results.Therefore, in one embodiment, the LPR system applies the aforementionedgeneral rules and trained models using artificial intelligence to selecta plurality of appropriate images from those captured and processed bythe LPR system. And, these selected images are merged into aconsolidated image. As a result, the characters of the license platehave a greater probability of being recognized by a computerized OCRplatform in the consolidated image than in any one of an initialshort-exposure image, a subsequent short-exposure image, an initiallong-exposure image, or a subsequent long-exposure image. Notably, themerging of the aforementioned images might not use a temporal noisefilter, as in some embodiments disclosed herein. Rather, the merging mayinvolve one, some, or all of the long-exposure images 1103 captured bythe camera device of a license plate. Alternatively, the merging mayinvolve one, some, or all of the short-exposure images 1105 captured bythe camera device of a license plate. In some instances, the mergingmight even involve selecting some long-exposure images (or other imagescaptured with adjusted settings) and some short-exposure images formerging into a consolidated image. The determination of whether toselect one image over another may include factors such as whether thecalculated relative speed of the target license plate is above or belowa threshold, whether the image is captured in a low-light situation, andwhether a sufficient quantity of images have been captured. Of course,in some embodiments, executing a temporal noise filter to generate aconsolidated image may be advantageous and may be incorporated into themethodology executed by the LPR system.

Finally, to perform OCRing of the plurality of the capturedmulti-exposures images and/or merged/consolidated images, the LPR systemmay feed the aforementioned images to an AI-trained model to performoptimal OCRing. The AI model may be resident at and executing on aprocessor, such as GPU 706 in FIG. 4 , at the vehicle, or may be locatedat a remote server 704 that is communicatively coupled with thecomponents in the vehicle. The OCR of the license may includeidentification of the characters and/or the state classification.Additional information may also be extracted from the license plateimage including but not limited to expiration date of the license platerenewal sticker and other information.

The second of four use cases describes one illustrative temporal noisefiltering (TNF) use case with following traffic (i.e., traffic that ismoving generally on the same roadway in the same direction as thevehicle equipped with the LPR system). The initial step in thisillustrative use case is similar to the steps and/or sub-steps describedin the first example use case above. As explained above, the LPR systemmay use one or more object detection libraries to find an object in acaptured image that matches the characteristics of a license plate. Thelibrary takes a captured image as input and identifies a license platein the image using object detection. As explained above, a heat map andprobability/confidence score may be generated using AI.

For example, the LPR system may comprise a tangible computer memory anda specially-programmed computer processor. The LPR system may, in someembodiments, include a camera device attached to a police vehicle. Thememory may store image data (e.g., long-exposure images, short-exposureimages, and other data) captured by the camera device, including a firstimage of the target vehicle at a first time and a second image of thesame target vehicle at a second time. The processor may receive thefirst image from the memory, where the first image shows the targetvehicle at a first position. Moreover, the processor may detect alicense plate in the first image, where the license plate is in a firstportion of the first image.

Next, a processor in the LPR system may use one or more object trackinglibraries, several of which are currently commercially available, todetect the same license plate in one or more subsequently capturedimages. The processor may seek out a second image from the memory, wherethe second image shows the target vehicle at a second position that isdifferent from the first position. The processor may predict the secondposition of the target vehicle based on the direction of the vehicleand/or the relative motion of the target vehicle between instances oftime. Alternatively, the processor may be programmed to seekcharacteristics of the vehicle (e.g., vehicle color, shape, make/model)to assist in identification of the same license plate. Moreover, theprocessor may simply detect the license plate in the second image, wherethe license plate is in a second portion of the second image.

In one example, the LPR system may demarcate a boundary of (e.g.,bounding box around) the license plate to track its position acrossframes. For example, the library may calculate a feature vector from thelicense plate and detect the feature vector in the subsequent image(s).In some examples, the tracking may be improved by increasing the framesper second (fps) capture rate. In one example, 60 fps may be beneficialfor high speed deltas. The number of frames captured can range from2-100 (or more). And the fps can be varied as appropriate. Theaforementioned settings may be adjusted either statically, dynamically,or manually, as the system is trained and optimal/desired settings areidentified for specific situations. In an alternate embodiment, thecamera device on the LPR system may generate a video feed comprising themultiple captured frames. The video feed may be regular (i.e., notcompressed) before OCRing; or, in other examples, the video feed may bea 4K30 HEVC compressed input or other compressed input. For example, theLPR system may comprise a video encoder configured to encode image datareceived by the image sensor in a format, such as MPEG-2 (H.262), MPEG-4(H.264), AOMedia Video 1 (AV1), etc.

In some examples, to conserve memory, the LPR system may, at some pointin time, discard all of the captured image outside of the bounding boxarea, which contains the pertinent license plate information. At leastone technological benefit of this step is that less memory is consumedbecause non-critical image data is discarded from memory. The processorof the LPR system may crop the first image to discard outside of thefirst boundary of the first image and crop the second image to discardoutside of the second boundary of the second image. In some examples,the detecting of the boundary (e.g., bounding box area) and subsequentcropping may be performed by a server computer with a high-speedprocessor (e.g., a GPU or a chipset that uses artificialintelligence/machine learning for detect operations). The server may belocated at the vehicle equipped with the camera device (e.g., in thetrunk of the vehicle), or the server may be located remote from thevehicle but communicatively coupled to the LPR system at the vehiclethrough wireless communication. Although communication with a remoteserver may introduce latency, thus delay, into the responsiveness of thesystem, the server may provide higher-speed processing of potentiallycomputationally intensive detection and tracking operations. In analternate embodiment, the on-premise processor may be configured toperform some or all of the aforementioned computations, but may offloadcomputations to a server when suitable—for example, during times wherethe on-premise processor is overloaded with high-priority computations.

To improve the probability of discerning the contents of the licenseplate, the processor may align the license plate in the first portion ofthe first image and the license plate in the second portion in thesecond image, as depicted in FIG. 7 . In the example 700 of FIG. 7 , thelicense plate is likely of a target vehicle that is following trafficbecause the image does not require much geometric rectification. Rather,FIG. 7 shows that the original captured images (as shown in the top twophotos denoted with “original frame”) is not aligned. The LPR systemdescribed herein aligns the images so that the position of the licenseplate in both images nearly matches the pixels of each image. In oneexample, the alignment may be performed by identifying a fixed pixel(e.g., upper-left corner of the overall license plate) of the image andaligning the image based on that fixed pixel. With the images aligned(as shown in the bottom two photos denoted with “Aligned+filtered”), afiltering process, such as TNF/TNR, may operate on the consecutiveimages to enhance the legibility of the alphanumeric characters (orother information) on the license plate. The filtering process may beperformed on simply two images, or may be performed on a plurality ofaligned images. For example, FIG. 7 shows the results 700 (i.e., theimage in the right-bottom corner) of aligning and filtering of threeimages. With an increased number of images and/or processing, thelegibility of the license plate information improves.

In addition to aligning the images of the license plates across frames,the processor of the LPR system may also transform the image to furtherenhance the legibility of the license plate information. For example, afirst portion and second portion of the image that encompasses thelicense plate may be further processed to optimize the legibility of thelicense plate information. The transforming may include geometricallyrectifying one or more frames to accommodate for relative positions ofthe target vehicle at a first position and a different second positionwhen subsequent images are captured by the camera device. The LPR systemmay use one or more commercially-available libraries that assist intransforming images, including scaling the image, warping the image,rotating the image, and/or other functions performed on the image. Oncethe images are geometrically rectified and aligned, the images are inoptimal condition for application of an image processing filter toenhance legibility of the alphanumeric information on a license plate.

In one example, once aligned and transformed, the processor of the LPRsystem may execute a temporal noise filter (TNF) on the first portion ofthe first image and the second portion of the second image to generate aconsolidated image. Referring to FIG. 9A, FIG. 9B, and FIG. 9C, theresults 900 of the filtering is that the consolidated image (e.g.,compare FIG. 9C to FIG. 9A) has a higher probability that thealphanumeric characters and images of the license plate in theconsolidate image are recognized by a computerized optical characterrecognition (OCR) platform than the license plate in the first image.With the filtering, as FIG. 9C illustrates, an availability of a largerquantity of consecutive images taken at different exposures (and othersettings) aids in providing a higher quality consolidated image. In oneexample, a consolidated image is considered to be higher quality whenmore of the characters on the license plate are correctly identified byan OCR platform. In another example, an LPR system is considered togenerate a higher quality consolidated image when fewer images are usedto generate a filtered end result that correctly results in an OCRplatform identifying the same number of characters on a license plate.At least one advantage of filtering is that it may result in charactersin the image having a sharper contrast and image. As illustrated in FIG.8 , an unfiltered image 802 produces characters that are less sharp andmore difficult to recognize than a filtered image 804.

Although this use case mentions aligning, transforming, and filtering ofimage frames to arrive at an optimized output image, this disclosurecontemplates and covers embodiments where one or more of the aligningand transforming steps are omitted. Although the resulting image may notbe of as high quality as compared to when all processing steps areperformed, alternative implementations may find such processingbeneficial—e.g., if a processor is overloaded or inaccessible and unableto perform all the aforementioned steps, or if a faster response time iscritical. In addition, although the aforementioned example references aprocessor at the vehicle executing the align, transform, and TNFfiltering steps, in some examples, the processing unit may be co-locatedbetween a first processor at the vehicle and a second processor in aserver machine (e.g., in a cloud environment readily accessible from thevehicle). In such embodiments, the processor at the vehicle may captureimages and perform some/no/little pre-processing of the captured images,then send the image to a processor in the server to perform additionalsteps of aligning, transforming, and/or application of a filter. Theprocessor in the server machine may also be responsible for calculatinga relative speed of the target vehicle based on a change in position ofthe license plate on the subsequently collected images. The disclosurecontemplates that in some scenarios locating the processor at thevehicle with the camera device performing the image capture may reducelatency and improve response time.

In a similar vein, the disclosure contemplates other examples involvingcombinations or sub-combinations of the aforementioned steps. In someexamples, the temporal noise filtering (TNF) may be applied to a subsetof all of the plurality of frames. In other examples, differentsub-combinations of the plurality of frames may be used until a bestfinal image is identified. Specifically, depending on the desiredresponse time/latency of the LPR system, the processor may selectspecific images for immediate processing on-site, while transmitting allor some of the image data to a server with a high-speed processor foradditional processing. The results of the two processes may be comparedand the on-site results may be supplemented/corrected if a more preciseOCR is performed by the server.

In another example, other filtering techniques besides TNF may be used.At least one advantage of TNF over traditional multi-frame noise filtersis that the shape and size of the moving license plate changesdramatically as it passes the camera device. The LPR system isconfigured with the information of the shape of the license plate (e.g.,rectangular) and uses this fact to further optimize the imageprocessing. In some examples, the LPR system uses the warped licenseplate stack to reduce the noise and improving its visual quality, e.g.,averaging or super-resolution. TNF is one of many potential methods thatmay be used to improve image quality of captured license plates. TNF isparticularly effective and provides better results when the same licenseplate is tracked and captured for multiple frames at different times,then aligned between frames. This disclosure contemplates that otherfiltering techniques or hybrid combination of filters may be used on theplurality of frame data.

Once the final output of the filtering stage is complete, theconsolidated image may be submitted to an OCR platform foridentification of the characters and/or the state classification of thelicense plate. As illustrated in FIG. 9C, the license plate informationappears to be “12999”. The LPR system may transmit this license plateinformation and/or other information to other systems for processing andevaluation, as described herein, for law enforcement purposes and/orother purposes.

The third of four use cases describes one illustrative temporal noisefiltering (TNF) use case with incoming traffic (i.e., traffic that isnot moving on the same roadway in the same direction as the vehicleequipped with the LPR system). Incoming traffic may be traffic that ison the same roadway as the vehicle equipped with the LPR system, butalso includes traffic that is on another roadway (e.g., an intersectingstreet, an adjacent high-way on-ramp, and others). The incoming traffichad a relative speed delta that attenuates the captured image more thanthe preceding illustrative use case involving following traffic.

In this illustrative use case, the initial steps are similar to thepreceding use case in that the LPR system captures images using one ormore camera devices, then license plate (LP) detection and trackingoccurs. However, because the images collected from incoming traffic tendto be more attenuated, the steps of optimizing the image by geometricrectification are more extenuated. For example, whentransforming/optimizing the image of the license plate using geometricrectification, the scaling, warping, rotating, and/or other functionsperformed on the image may be extenuated because both the relative speeddelta may be higher and the angular speed of the incoming vehicle willincrease as the vehicle gets closer.

FIG. 10 illustrates that a car's license plate 1002 is detected in theimage, then edge detection is performed on the image 1004 of the licenseplate. Although the car in the image 1002 is following traffic, forvarious reasons, the image of the license plate is not the traditionalrectangular boundary shape. Rather, the image 1004 is a parallelogramboundary shape. The LPR system detects that the image requirestransforming and executes one or more libraries on the image to adjustthe image into the transformed image 1006. The transformed image, ifsufficiently sharp in some examples, might not require aligning or TNFprocessing. Rather, in that example, the transformed image may be sentto an OCR platform for immediate processing and determination of thelicense plate information. In other examples, multiple images of thealigned, transformed image may be applied to a TNF to generate a higherquality composite image. In addition to determining the license platenumbers, the LPR system may also analyze characteristics of the licenseplate image 1008 to identify the state/country of the plate. In thiscase, the formatting of the alphanumeric license plate numbers, colors,and positioning, along with preliminary image recognition of the“Illinois” in the upper portion of the license plate image, permits theLPR system to determine with a particular confidence score that this isan Illinois license plate. In addition to image processing, the LPRsystem may also consider GPS coordinates, as provided by a locationdetermination unit (e.g., GPS receiver) in the camera vehicle, to narrowthe list of likely state plates. For example, any given vehicle locatednear the border of Illinois and Indiana has a higher likelihood of beingan Illinois or Indiana plates than Georgia plates. The LPR system mayaccess rules and ML-trained models regarding state plates to increasethe accuracy of its determination.

In the preceding example, a TNF is used to sharpen the characters of thelicense plate in the one or more images. Temporal noise filtering isdifferent from traditional multi-frame noise filters because, amongother things, the shape and size of the boundary of the moving licenseplate changes dramatically as it passes the camera car. The LPR systemis able to detect, track, and then transform the license plate image byusing the fact that the shape of the license plate is known andpredefined. In some examples, the LPR system may use the warped licenseplate stack to reduce the noise and improving its visual quality. e.g.,averaging or super-resolution. In some implementations, the LPR systemmay supplement or supplant the processor with an application-specificintegrated circuit (ASIC) processor. The ASIC processor is designed toperform the specific operations and functionality described herein, thusproviding a potentially faster response time and computational savings.

In some examples, the processing unit (e.g., processor) may beco-located between a first processor at the vehicle and a secondprocessor in a server machine (e.g., in a cloud environment readilyaccessible from the vehicle) to distribute execution of the align,transform, and TNF filtering steps. The processor at the vehicle maycapture images and perform some/no/little pre-processing of the capturedimages, then send the image to a processor in the server to performadditional steps of aligning, transforming, and/or application of afilter. In a similar vein, the disclosure contemplates other examplesinvolving combinations or sub-combinations of the aforementioned steps.In some examples, the temporal noise filtering (TNF) may be applied to asubset of all of the plurality of frames. In other examples, differentsub-combinations of the plurality of frames may be used until a bestfinal image is identified. Specifically, depending on the desiredresponse time/latency of the LPR system, the processor may selectspecific images for immediate processing on-site, while transmitting allor some of the image data (e.g., long-exposure images, short-exposureimages, and other data) to a server with a high-speed processor foradditional processing. The results of the two processes may be comparedand the on-site results may be supplemented/corrected if a more preciseOCR is performed by the server.

This use case contemplates and covers embodiments where one or more ofthe aligning and transforming steps are omitted. Although the resultingimage is not of as high quality as compared to when all processing stepsare performed, alternative implementations may find such processingbeneficial—e.g., if a processor is overloaded or inaccessible and unableto perform all the aforementioned steps, or if a faster response time iscritical.

Referring to FIG. 13 , the fourth of several use cases describes oneillustrative embodiment with aspects of multi-exposure capturingfeatures coupled with temporal noise filtering (TNF). The initial stepin this illustrative use case is similar to the steps and/or sub-stepsdescribed in the three example use cases above. As explained above, theLPR system may use an image sensor to take multiple images of the samelicense plate, but under varying conditions. The camera device keepsseparate two streams of image frames being captured by the image sensor.The image sensor may be an HDR sensor in some examples. A high dynamicrange (HDR) image is captured by taking multiple photos of the samelicense plate, but each at different shutter speeds, thus resulting inimages with varying brightness/shadows/highlights: bright, medium, anddark. The image brightness is based on the amount of light that arrivesat the image sensor. The HDR sensor includes post-processingcircuitry/firmware that combines the series of images by adjusting thecontrast ratios to bring details to the shadows and highlights; theresulting consolidated image is usually not possible with a singleaperture and shutter speed. The consolidated image is made by takingmultiple images of the same scene, but each at different shutter speeds,resulting in a bright, medium, and dark images based on the amount oflight that gets to the lens.

The term HDR image sensor, as used in this disclosure includes but isnot limited to a dynamic range sensor, a wide dynamic range (WDR)sensor, and other sensor types. In some examples, a wide dynamic range(WDR) sensor provides dual-exposure (dark and light) image/video capturethat when consolidated into a composite image, is able to retain detailsin both light and dark portions of the frame. This keeps bright areasfrom looking over-exposed and darker areas from losing detail inhigh-contrast situations. Moreover, modern image sensors can sometimescapture a high dynamic range from a single exposure. The wide dynamicrange of the captured image is non-linearly compressed into a smallerdynamic range electronic representation. However, with properprocessing, the information from a single exposure can be used to createan HDR image. Other types of image sensors as also contemplated for usein this disclosure, including but not limited to charge-coupled device(CCD) sensors, complementary metal-oxide (CMOS) sensors, and organicphotoconductive film (OPF) sensors, i.e., a type of imaging sensor thatuses two separate layers—one that's the light-sensitive “film” andanother layer of circuits—to transform that light layer into electricalcurrents to create a digital image. OPF sensors are sometimes better inlow light because of that multilayer design; the layer structuresometimes allows division of the pixel's electrodes into large and smallareas such that the image sensor can then change the voltage applied tothe first layer, essentially adjusting how sensitive the sensor is tolight on a per-pixel basis. The effect is a wider dynamic range.

Referring to FIG. 11 , an image sensor may capture images at a rate of anumber of frames per second (e.g., 60 fps). In one example, an imagesensor may be set to alternate exposure every other frame to capture aplurality of images 1101. The captured images 1101 may be split intoseparate streams of long-exposure images 1103 and short-exposure images1105. In effect, this results in a halved frame per second rate (e.g.,30 fps) for each of the image streams. The image sensor in theaforementioned example is nearly simultaneously outputting images of thefield of view in a serial manner, e.g., frame by frame. The setting ofthe camera device is alternated every other frame from the long-exposuresetting to the short-exposure setting, even when the image sensor is asingle image sensor.

In contrast, referring to FIG. 11 , The image sensor may nearlysimultaneously output images 1106 of the field of view in a parallelmanner, e.g., line by line. The image sensor may be set to capture witha long-exposure setting for a first set of lines in a frame whilesimultaneously to the short-exposure setting for a second set of linesin the same frame. As a result, multiple exposures can be taken inparallel (e.g., line by line using line pairs on Bayer mosaic) within asingle frame, and then the exposures can be separated; the separatedstreams may result in lower resolution in each of the separate streamsof long-exposure images 1108 and short-exposure images 1110. AlthoughFIG. 11 shows the plurality of frames 1106 with an interlacedline-by-line configuration, the first set of lines with a first exposure(or other) setting and a second set of lines with a second exposure (orother) setting need not be in an interlaced configuration. For example,the image sensor may be set to capture with a long-exposure setting fora first set of pixels on a sensor, and set to simultaneously capture ashort-exposure setting for a second set of pixels on a sensor, whereinthe first set of pixels is different than the second set of pixels(e.g., alternating in a checkerboard pattern). Moreover, the separatestreams 1108 and 1110 (and for that matter, streams 1103 and 1105) neednot be just different exposure settings. In some examples, othersettings of the image sensor, as described throughout this disclosure,may be adjusted to provide a variation in image capture.

In this use case, the LPR system may calculate the relative speed of thevehicle with a license plate in the captured images. Some of thechallenges in using a camera to capture license plates of vehicles isfast relative motion, potential low light conditions, and a combinationof both. To enhance license plate recognition, the camera device mayincrease the duration of time the shutter is open (i.e., slower shutterspeed), but this can cause more blur when objects (e.g., vehicles) aremoving fast relative to each other. In order to reduce blur and optimizelicense plate capture, the relative speed of the vehicles may becalculated using video analytics of a detected license plate—e.g.,optimize shutter speed in subsequent moments to capture images with themost light possible given the speed of relative movement. In oneexample, the relative speed may be calculated using motion blur analysisof a frame. In another example, the relative speed may be calculatedusing one or more of the methods disclosed herein in combination withone or more hardware components disclosed herein. Using the calculatedrelative speed, the LPR system may optimize the exposure (or other)settings on one or both the long-exposure setting and short-exposuresetting of the image sensor.

For example, referring to FIG. 13 , the exposure settings of the firststream may be optimized for detecting plates on vehicles with a lowrelative speed (in step 1302), while the second stream may be optimizedfor detecting plates on vehicles with a high relative speed (in step1304). In general, when using a single camera to capture multipleexposure images, there is a greater likelihood of OCR of license platesof oncoming traffic (i.e., those with a greater relative speed) in theshort-exposure images. Meanwhile, there is a lower likelihood of OCR oflicense plates of oncoming traffic in the long-exposure images. And, alower likelihood of OCR of license plates of oncoming traffic inconsolidated images, such as the image resulting from the merging of thelong-exposure image with the short-exposure image because such mergingmay introduce blur. Rather, the LPR system disclosed herein, in someexamples, increases likelihood of OCR by iteratively cycling throughexposure times (effectively capturing more exposures). In one example,the LPR system may use a controller (or other component) to adjustexposure settings by a predetermined amount (e.g. 1.5×). For example,for exposures captured in series, theist frame may be 1 ms (firstshort-exposure frame), 2^(nd) frame may be 10 ms (first long-exposureframe), 3^(rd) frame may be 1.5 ms (second short exposure frame), andthe 4^(th) frame may be 15 ms (second long-exposure frame). Theaforementioned values are merely one example, and the values in variousembodiments may be different and varied as appropriate. In someexamples, the LPR system may cycle through the exposure settings, or mayuse a random shuffle of the values to capture varying exposure ofimages. In another example involving capturing in parallel, the 1^(st)frame may be at 1 ms for a first group of lines and 10 ms for a secondgroup of lines, and the 2^(nd) frame may be at 1.5 ms and 15 ms. Theaforementioned values are merely one example, and the values in variousembodiments may be different and varied as appropriate. Othercombinations and variations of the predetermined values, randomshuffling, and other methods, as disclosed herein, are contemplated inthe aforementioned examples.

In some examples, the LPR system may optimize/adjust image sensorsettings (e.g., exposure) based on detected speed of the tracked licenseplate. Then the calculated relative speed may be used to tune thecamera's exposure settings to best capture the plate for opticalcharacter recognition. Referring to FIG. 13 , in step 1306 and 1308, theLPR system captures additional images with the adjusted long-exposureand short-exposure settings. In some examples, the exposure settings mayhave been set based on lighting conditions. In some embodiments, ahybrid model may be used that takes a weighted combination (e.g., 50/50equal weight, or some other weight) of relative speed and lightingconditions to determine and set optimal camera device settings. Forexample, the combination may be a subset of claimed features or part ofthe overall system claimed. In some examples, the properties of thecapturing camera device may be adjusted automatically or dynamically.Moreover, in some examples, the exposure times may be selected byauto-exposure so that the camera device best balances the motion blurand noise for the image streams. Finally, in some examples, thelong-exposure stream may be optimized for following traffic and theshort-exposure time for the oncoming traffic. For example, in FIG. 6B,with respect to vehicle 607, the system detects a license plate in afirst portion 601 of the first short-exposure time image that wascaptured with a first short-exposure setting. Then, at a later time, theLPR system captures a second short-exposure time image, as illustratedin FIG. 6C, and detects the same license plate in a first portion 601 ofthe second short-exposure time image. The second short-exposure timeimage may be captured with the same or different settings as the firstshort-exposure setting. In one example, the second short-exposuresetting may have a fast shutter speed and increased ISO (to maintaintotal exposure of image) compared to the first short-exposure setting.

Referring to FIG. 13 , next, the long-exposure and short-exposure imagestreams may be modified as described in the preceding examples inpreparation for temporal noise filtering (TNF). In step 1310, the LPRsystem detects the license plates in the images. And, in step 1312, theLPR system determines if the license plate is on following traffic oroncoming traffic. If it is oncoming traffic, then the stream ofshort-exposure images is aligned and transformed in steps 1314 and 1316,as described below. Meanwhile, if it is following traffic, then thestream of long-exposure images is aligned and transformed in steps 1318and 1320. In one example, only the long-exposure images may be alignedand geometrically rectified before executing a TNF on the multipleimages to create a single, composite image in step 1322. In anotherexample, the same steps are performed on only the short-exposure images.In yet another example, the long-exposure images and short-exposureimages may be mixed and the license plate may be tracked across the two,separate image streams to generate the best TNF, composite image. Inanother example, the system may calculate the angular velocity of theincoming traffic and select between the long-exposure and short-exposurestream accordingly. In other words, one or both of the images from eachtimestamp may be selected for TNF before consolidation into a singleimage. The LPR system applies temporal noise reduction/filtering (TNF)to the plurality of images, such as the aforementioned ones referencedin FIG. 6B and FIG. 6C, by aligning, transforming, and/or merging into aconsolidated image. The consolidated image may be OCRed to identify thelicense plate characters and other identifying information.

In some examples, the LPR system may further comprise a locationtracking device coupled to the camera device. The processor of the LPRsystem may be programmed to stamp an image with a location of the cameradevice at the time when the image is captured by the camera device. Inaddition, the LPR system may also comprise a clock mechanism. Theprocessor of the LPR system may be programmed to timestamp an image uponcapture by the camera device. At least one benefit of the aforementionedmetadata associated with the captured and processed image is thatevidentiary requirements in a legal proceeding or other investigationmay be satisfied. Moreover, for report generation purposes, themetadata, e.g., location, date, time, and other information, may becollected into a central data store, indexed, tagged, and displayed in avisually useful format. In addition, the tracking of license plates canalso produce other useful information, e.g., how fast the cars aremoving. And, numerous actions can be triggered based on this usefulinformation. In one example, the LPR system measures the relative speedof the vehicles using video analytics of the recognized license plate.Then, the system optimizes shutter speed in subsequent moments tocapture images with the most light possible given the speed of relativemovement.

As explained above, in one example the camera device attached to thepolice vehicle may include a plurality of cameras arranged at differentlocations of the police vehicle and configured to operate in acoordinated manner to capture images of vehicle license plates or otheritems. The captured images may be output to a shared memory. A computerprocessor of the LPR system may receive from the memory one or more ofthese images captured from multiple cameras, and then perform one ormore of the methods disclosed herein. In one example, a single lawenforcement vehicle may be equipped with one camera device facingtowards traffic in front of the vehicle and a second camera devicefacing towards traffic to the rear of the vehicle. In another example,an additional camera device may be positioned to the right or left sideof a law enforcement vehicle to assist in capturing license plate imagesof vehicles traveling at an angle (e.g., perpendicular at a streetintersection) to the law enforcement vehicle. The image sensors fromthese multiple camera devices may capture images and process thecollective images to identify the characters and other characteristic oflicense plates of vehicles in their proximity. The processor of the LPRsystem may use one or more images, which are stored in the sharedmemory, from each of the camera devices to increase the probability ofrecognizing by a computerized optical character recognition platform,the characters of the license plates. In some embodiments, multiplecamera devices may be affixed to a single vehicle in variousorientations and/or positions. In addition, in some examples, at leastone of the aforementioned plurality of cameras may include an unmannedaerial vehicle (UAV) equipped with video capture capabilities. The UAVmay be mounted to the vehicle and may be automatically launched asappropriate by the LPR system upon occurrence of particular triggerevents.

The system is not limited to traditional vehicles. Rather, unmannedaerial vehicles (UAVs) or drones are also considered vehicles forpurposes of this disclosure. FIG. 2B illustrates a UAV equipped with thedevice 201. The installation of the device 201 on a UAV may rely uponcomponents that were optional in a car installation. For example, GPSunit 212 (or comparable location tracking technology) may be critical toa device 201 installed on a UAV because of the UAVs ability to traveloutside the confines of traditional lanes on a highway. Moreover, theUAV may optimize the illumination pattern from the device 201 to focusin a downward direction towards the road. The micro-controller 204 andAI component 216 in the device 201 may be used to train the system tooptimize the capturing of license plate numbers of vehicles. Finally, inUAV installations, the operations of the camera apparatus 201 may bemodified to account for any high-frequency vibrations that might occurfrom capturing images or video from a UAV. For example, a global shutterfeature may be implemented in the camera apparatus 201 to reduce rollingshutter effect that might be caused by vibrations.

Referring to FIG. 2C and FIG. 2F, the illustrations are of embodimentsof the system for commercial sale and use. Assembly 201 is aconfiguration of a camera apparatus 201 with a plurality of lightemitting apparatuses 230. The assembly 201 may be mounted to the frontof a police car to capture images for license plate recognition. Theassembly 201 may draw power from the subject vehicle. Although FIG. 2Cdepicts the components of assembly 201 as a single object, in someexamples, the parts of assembly 201 may be separated and installedseparately or in an organized, different manner.

FIG. 2F illustrates one installation of the system where just one lightemitting apparatus 230 is combined with the camera apparatus 201. Thedevice may be mounted inside a vehicle, or outside a vehicle. Forexample, as illustrated in FIG. 3 , the device 201 may be mounted to thetop of a police car 108. Moreover, the device may be capable ofover-the-air (OTA) software updates to its software and data. The device201 may also seamlessly connect with a local, wireless networkcomprising other components in the vehicle 108 and accessories 322carried by or worn on the operator of the vehicle 108. Moreover,connecting with the local network provides the device 201 with eventnotifications, such as when the operator opens the car door, activates apolice car's light/siren bar, and other events, so the device 201 canreact accordingly. Once connected with the local network of devices, thedevice 201, 412 may connect as illustrated in FIG. 4 with a computingdevice 414 to assist it with calculations, storage, and othercomputations.

Although the grid pattern 230 in FIG. 2E is illustrated as a rectangulararrangement, the configuration of light sources (e.g., LEDs) in a gridis not limited to such arrangements. The grid may be a circular grid,elliptical grid, or any other shape conducive to generation of anillumination pattern. The mounting apparatus may operate in response toan electrical signal received from a micro-controller or other externalsource. As a result, the light emitting apparatus 230 may generate acustomized illumination cone as desired. Although LEDs are used as oneexample of a light source, other types of lighting elements, such ashalogen bulbs and the like are contemplated for use with the system.LEDs, however, may be preferred due to their fast response time, abilityto be switched on and off at a high frequency without substantiallyimpacting bulb life, and lower power consumption. In some examples, theLEDs and other light sources may emit light in the infrared frequencyrange to aid in image capture in low-light or night time situations.Another benefit of infrared light is that it select frequencies ofinfrared light are non-visible to the eye, thus has a less negativeimpact on operators of target vehicles in oncoming traffic. Furthermore,infrared light may be desirable in situations where covertness isdesired. For example, the LEDS and other light sources may emit infraredlight comprising a peak wavelength of between 800 nanometers and 850nanometers, between 850 nanometers and 900 nanometers, between 900nanometers and 950 nanometers, between 950 nanometers and 1000nanometers, between 1000 nanometers and 1100 nanometers, or between 800nanometers and 1100 nanometers according to various embodimentsdescribed herein.

Referring to FIG. 2E, a standalone light emitting apparatus 230 isillustrated. The light emitting apparatus 230 may include amicro-controller 204, similar to the one in the camera apparatus 201,for controlling configuration settings of the light emitting apparatus230. The light emitting apparatus 230 provides functionality to thesystem because it generates and emits the illumination pattern thatimproves image capture quality. The light emitting apparatus 230 may bemounted to a vehicle such as a police car, motorcycle, or other vehicle.The apparatus 230 may be mounted inside the vehicle, or may be securelymounted to the outside of the vehicle. The light emitting apparatus 230comprises a body, at least one light source, a mounting apparatus 232inside the body that couples the light source to the body, and amicro-controller. As illustrated in FIG. 2E, the mounting apparatus 232may be coupled with the light source such that it permits themicro-controller to automatically, and without human intervention, tiltthe light source along at least one of its roll, pitch, and yaw axes. Insome examples, the mounting apparatus might allow adjustment in allthree axes in response to a tilt command from the micro-controller. Theend result of the tilting and re-orienting of the light sources is anasymmetrical illumination cone pattern being emitted towards a lane nearthe one on which the subject vehicle is traveling. The target vehicle'slane need not necessarily be adjacent to the subject vehicle's lane.Rather, the system may be trained to adapt to different roadconfigurations in different geographic areas.

In addition to tilt commands, the micro-controller may also generate andsend illumination commands to the light source. The light source may befurther configured to emit light at one of a low, medium, and highillumination in response to an illumination command Illuminationcommands are not limited by the enumerated list provided here. Rather,illumination commands may include any denotation of varying illuminationlevels.

Whether a light emitting apparatus 230 will emit low, medium, or highillumination is based on the values generated by the distancemeasurement component and the speed delta measurement component. In oneexample, the distance measurement component and the speed measurementcomponent may share a laser beam generator positioned in the body. Thelaser beam generator is configured to emit a laser beam to measure theapproximate distance to the target vehicle and the relative speed of thetarget vehicle. Such measurements are then sent to the micro-controllerfor rapid decision making. In an alternate embodiment, an externaldevice may provide tilt commands and illumination commands through anexternal port interface in the light emitting apparatus 230.

Regarding FIG. 14 , the flowchart illustrates an example of an operationof a light emitting apparatus 230 in accordance with various aspects ofthe disclosure. In steps 1402 and 1404, the system measures 1402 anapproximate distance to a target vehicle in a lane near one on which thesubject vehicle is traveling. The approximate distance may be calculatedusing a radar system as disclosed herein. The system also calculates1404 a relative speed of the target vehicle in the lane relative to aspeed of the subject vehicle in its own lane. The relative speed (or thespeed delta) may also be calculated using a radar system as disclosedherein. In some examples, these two values are inputted into amicro-controller in the light emitting apparatus 230. Alternatively, themicro-controller 404 may receive raw input and calculate the speed deltavalue and distance value itself. Alternatively, an external source maycalculate these values and provide them to the apparatus 230 through anexternal interface, such as a physical port or through a wirelessantenna.

Next in step 1406, the micro-controller may generate a tilt commandand/or an illumination command based on the received inputs. Thecommands may be sent 1408 to their respective destinations: the tiltcommand is sent to the mounting apparatus 232 to effect a change inorientation of the emission of the one or more light sources attached tothe light emitting apparatus 230. Meanwhile, the illumination commandmay be designated with one of several values. See step 1410. In oneexample, the illumination command values may be from the enumerated listof low, medium, or high. Based on the value, the LED light source mayemit a low illumination 1412, medium illumination 1414, or highillumination 1416. For example, the micro-controller 234 may send anapproximate voltage level to the light source in the light emittingapparatus, corresponding to the low value, medium value, or high valueof the illumination command. As a result, the light source may emit abrightness of illumination corresponding to the approximate voltagelevel. The light emitted by the LED may be in an infrared frequencyrange and create an asymmetrical illumination pattern towards a lanenear to the one on which the vehicle is traveling.

In step 1418, in examples where the light emitting apparatus 230 isexternal to the camera apparatus 201, the light emitting apparatus 230and the camera apparatus 201 are synchronized by communications with orrelated to the operational state of each apparatus 201, 230. Theapparatuses may communicate directly, or they may communicate with acentral mediator or gateway device that controls their operation. Asillustrated in FIG. 4 , the camera assembly, which comprises the lightemitting apparatus and the camera apparatus, may communication withother devices on the network.

Regarding FIG. 15 , the graph depicts the relationship 1500 betweenspeed delta values (represented by the Y-axis) and distance values(represented by the X-axis) as they define the operation of an exemplaryembodiment of the disclosed system related to an ALPR system. Thecomponents of the system may include, in some embodiments, a cameraapparatus, a light emitting apparatus, a networked computercommunicatively connected via a wireless network with components in thecamera apparatus, and/or a vehicle include its telematics sensors,on-board diagnostics (OBD) port, and other components. Theseinterconnected components may coordinate to determine the speed deltavalue and distance values for any given image capture.

As the graph 1500 illustrates, the autonomous operation of the systemmay be programmed to operate under the scenarios described in FIG. 15 .The components of the system provide one or more of the inputs requiredby graph 1500. In some examples, the intermediate value may be a fixed,pre-set value. In other examples, the system may operate a feedbackloop, as illustrated in FIG. 5 to regularly update the intermediatevalue to optimize the operation of the system. Meanwhile, software,firmware, or hardware at the vehicle onto which the camera assembly ismounted, may control the settings in accordance with FIG. 15 . In someexamples, the camera assembly may simply include a computer-executablecode that executes on a processor in the camera assembly.

Some illustrative settings of the camera assembly include, but are notlimited to, exposure time, illumination power, focus position, sensorgain (e.g., camera ISO speed), aperture size, filters, and the like. Ingraph 1500, values for the exposure time and illumination power areillustrated for different operating scenarios. Scenarios A, B, C, and Dillustrated in counter-clockwise direction in the graph 1500 starting onthe lower-right, are described in more detail in relation to FIG. 2D. Aperson of skill in the art will appreciate that one or more of thesettings may be interrelated or dependant. For example, an exposure of1/25 sec at f/11, ISO 100 is equivalent to an exposure of 1/400 sec atf/2.8, ISO 100. In other words, because the shutter speed has beenreduced by four stops, this means less light is being captured by theimage sensor in the camera assembly. As a result, the aperture isincreased in size by four stops to allow more light into the cameraassembly. While there are benefits and disadvantages to adjusting thesettings in one way versus another, such knowledge would fall within therealm of a person having skill in the art. For example, a person havingskill in the art would understand that to maximize exposure, a cameraassembly might be set to a large aperture, 6400 ISO, and a slow shutterspeed. Meanwhile, to minimize exposure, a camera assembly would be setto a small aperture, 100 ISO, and a fast shutter speed. Of course, thesharpness of the captured image might be effected by depth of field,aperture, and shutter speed settings. In particular, with mostembodiments disclosed herein involving a moving subject vehicle capturean image of a moving target vehicle, the ability to capture an imagewithout introducing blurriness or shading or planar warp is aconsideration.

Moreover, in practice, target vehicles (e.g., oncoming traffic) on aroadway 102 traveling in a direction opposite to a subject vehicle onthe roadway 104 may be traveling at different speeds and be at differentdistances, as illustrated in FIG. 1A. Meanwhile, the continuous flow ofnew target vehicles on the roadway 102 (e.g., oncoming traffic andfollowing traffic) adds complexity to image capture. The optimum valuefor camera settings for each scenario is a non-linear function whichdepends on the camera performance parameters and detection algorithmsMeanwhile, in practice, a relationship such as depicted in FIG. 15 , mayprovide images of sufficient quality to capture objects and licenseplate numbers.

In addition to optimizing camera settings, the disclosed systemcontemplates a light emitting apparatus 230 coupled to the operation ofa camera apparatus 201 to further optimize image capture. FIG. 1Billustrates one example of an asymmetrical illumination pattern createdby the light emitting apparatus. In contrast, the illumination patternin a different example embodiment has been further optimized so thatmost intensive illumination is optimized to the edges of the camerafield of view. Meanwhile, the illumination straight ahead of the camerafield of view is reduced. Further optimization of the asymmetricillumination cone may be performed to strengthen the illumination on theleft-hand side lane, in the United States, because the oncoming trafficis there. Adding extra illumination in that direction enhances imagecapture because the relative velocity is higher and requires strongerlighting. Meanwhile, in scenarios such as FIG. 1A, which corresponds toroadways in countries like Britain and India, the oncoming traffic is onthe right-hand side lane; thus, the configuration set for theimplementation would be reversed for those countries. In yet anotherexample, the illumination cone may be a single illumination conepointing in the same direction as the camera, however, such an examplehas not been fully optimized for image capture. This is because the coneof the illumination and the camera field of view point to the samedirection; this approach is not optimal in some embodiments because itilluminates the lane in front of the car the best instead of theadjacent lanes.

In some examples, the light emitted by the disclosed system may beadjusted to further refine the illumination cone 108. In one example,the light emitting apparatus 230 may comprise a plurality of lightemitting diodes (LED) oriented in a grid pattern. Each LED may becoupled to a mounting apparatus that allows each individual LED to bere-oriented as desired by the system. For example, each LED may beseparately attached to a motorized mechanism (or other mechanism) toallow the system to roll, pitch, and/or yaw the LED as appropriate. Insome examples, a group of LEDs may be attached to a single mountingapparatus, thus they may all re-orient in unison. The grouping of LEDsmay be by column, by row, or by an area of the grid. Moreover, some LEDsmay be tilted in one direction, but others are tilted in a differentdirection. Thus, the illumination pattern may be tailored and tested toidentify an optimal configuration in particular geographic locations andenvironments. In yet another example, the plurality of LEDs in the lightemitting apparatus may be individually controlled. For example, one ormore LEDs may be sub-grouped and activated or deactivated together so asto emit an illumination pattern where the strongest light is pointed tothe edge of the camera field of view 106. As used in this example,activated and deactivated includes not only turning on and turning offan LED, but also dimming the illumination intensity of an LED.Alternatively, the illumination pattern may be such that the strongestlight is directed to oncoming traffic 108. The aforementioned sub-groupsmay be static, non-movable LEDs without a mounting apparatus toeffectuate tilting; alternatively, the mounting apparatus and sub-groupactivation/deactivation feature may be used together. The sub-groups maybe a plurality of LEDs in a single row or single column, or may beanother pattern of LEDs (e.g., diagonal line, circular pattern,semi-circular pattern, elliptical pattern, or other pattern).

Many alternatives to the systems and devices described herein arepossible. Individual modules/components or subsystems can be separatedinto additional modules/components or subsystems or combined into fewermodules/components or subsystems. Modules/components or subsystems canbe omitted or supplemented with other modules/components or subsystems.Functions that are indicated as being performed by a particular device,module/components, or subsystem may instead be performed by one or moreother devices, modules/components, or subsystems.

Although some examples in the present disclosure include descriptions ofdevices comprising specific hardware components in specificarrangements, techniques and tools described herein can be modified toaccommodate different hardware components, combinations, orarrangements. Further, although some examples in the present disclosureinclude descriptions of specific usage scenarios, techniques and toolsdescribed herein can be modified to accommodate different usagescenarios. Functionality that is described as being implemented insoftware can instead be implemented in hardware, or vice versa.

Many alternatives to the techniques described herein are possible. Forexample, processing stages in the various techniques can be separatedinto additional stages or combined into fewer stages. As anotherexample, processing stages in the various techniques can be omitted orsupplemented with other techniques or processing stages. As anotherexample, processing stages that are described as occurring in aparticular order can instead occur in a different order. As anotherexample, processing stages that are described as being performed in aseries of steps may instead be handled in a parallel fashion, withmultiple modules/components or software processes concurrently handlingone or more of the illustrated processing stages. As another example,processing stages that are indicated as being performed by a particulardevice or module may instead be performed by one or more other devicesor modules/components.

In this description herein of the various embodiments, reference is madeto the accompanying drawings, which form a part hereof, and in which isshown by way of illustration, various embodiments of the disclosure thatmay be practiced. It is to be understood that other embodiments may beutilized. A person of ordinary skill in the art after reading thefollowing disclosure will appreciate that the various aspects describedherein may be embodied as a computerized method, system, device, orapparatus utilizing one or more computer program products. Accordingly,various aspects of the computerized methods, systems, devices, andapparatuses may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects. Furthermore, various aspects of the computerizedmethods, systems, devices, and apparatuses may take the form of acomputer program product stored by one or more non-transitorycomputer-readable storage media having computer-readable program code,or instructions, embodied in or on the storage media. Any suitablecomputer readable storage media may be utilized, including hard disks,CD-ROMs, optical storage devices, magnetic storage devices, and/or anycombination thereof. In addition, various signals representing data orevents as described herein may be transferred between a source and adestination in the form of electromagnetic waves traveling throughsignal-conducting media such as metal wires, optical fibers, and/orwireless transmission media (e.g., air and/or space). It is noted thatvarious connections between elements are discussed in the followingdescription. It is noted that these connections are general and, unlessspecified otherwise, may be direct or indirect, wired or wireless, andthat the specification is not intended to be limiting in this respect.

In general, functionality of computing devices described herein may beimplemented in computing logic embodied in hardware or softwareinstructions, which can be written in a programming language, such asbut not limited to C, C++, COBOL, JAVA™, PHP, Perl, Python, Ruby, HTML,CSS, JavaScript, VBScript, ASPX, Microsoft.NET™ languages such as C#,and/or the like. Computing logic may be compiled into executableprograms or written in interpreted programming languages. Generally,functionality described herein can be implemented as logic modules thatcan be duplicated to provide greater processing capability, merged withother modules, or divided into sub modules. The computing logic can bestored in any type of computer readable medium (e.g., a non-transitorymedium such as a memory or storage medium) or computer storage deviceand be stored on and executed by one or more general purpose or specialpurpose processors, thus creating a special purpose computing deviceconfigured to provide functionality described herein.

Aspects of the invention have been described in terms of illustrativeembodiments thereof. Numerous other embodiments, modifications, andvariations within the scope and spirit of the appended claims will occurto persons of ordinary skill in the art from a review of thisdisclosure. For example, one of ordinary skill in the art willappreciate that the steps illustrated in the illustrative figures may beperformed in other than the recited order, and that one or more stepsillustrated may be optional in accordance with aspects of the invention.Moreover, the foregoing description discusses illustrative embodimentsof the present invention, which may be changed or modified withoutdeparting from the scope of the present invention as defined in theclaims. Examples listed in parentheses may be used in the alternative orin any practical combination. As used in the specification and claims,the words ‘comprising’, ‘including’, and ‘having’ introduce anopen-ended statement of component structures and/or functions. In thespecification and claims, the words ‘a’ and ‘an’ are used as indefinitearticles meaning ‘one or more’. When a descriptive phrase includes aseries of nouns and/or adjectives, each successive word is intended tomodify the entire combination of words preceding it. While for the sakeof clarity of description, several specific embodiments of the inventionhave been described, the scope of the invention is intended to bemeasured by the claims as set forth below.

The invention claimed is:
 1. A license plate recognition (“LPR”) systemattached to a law enforcement vehicle, the system comprising: a cameradevice configured to capture images; a memory, which is communicativelycoupled to the camera device, configured to store the images, whereinthe images comprise a first image of a license plate at a first time anda second image of the license plate at a second time; a processor, whichis communicatively coupled to the memory, programmed to: receive thefirst image from the memory, wherein the first image shows the licenseplate at a first position; detect the license plate in the first image,wherein the license plate is in a first portion of the first image;receive the second image from the memory, wherein the second image showsthe license plate at a second position that is different from the firstposition; detect the license plate in the second image, wherein thelicense plate is in a second portion of the second image; align thelicense plate in the first portion and the license plate in the secondportion; transform the first portion and the second portion bygeometrically rectifying to accommodate for relative positions of thelicense plate at the first position and the second position, wherein thegeometrically rectifying comprises scaling, warping, or rotating thefirst portion and the second portion; and execute, after thetransforming, a temporal noise filter on the first portion of the firstimage and the second portion of the second image to generate aconsolidated image, wherein the consolidated image has a greaterprobability that characters of the license plate in the consolidatedimage are recognized by a computerized optical character recognitionplatform than the license plate in the first image.
 2. The system ofclaim 1, further comprising: a controller, which is communicativelycoupled to the camera device, configured to modify a capture setting ofthe camera device; and wherein the processor is further programmed to:calculate a relative speed of the license plate based on the relativepositions of the license plate at the first position and the secondposition; and instruct the controller to adjust the capture setting ofthe camera device based on the relative speed, such that a third imageis configured to be captured with a different capture setting than thefirst image.
 3. The system of claim 1, further comprising: a wirelesscircuitry configured to receive a command from an external system,wherein the command causes the camera device to capture the image,wherein the external system comprises at least one of a remote commandcenter, another law enforcement vehicle, and a body-camera device. 4.The system of claim 1, further comprising: a controller communicativelyconnected to the camera device, wherein the controller is configured tomodify a shutter speed setting of the camera device; and wherein theprocessor is further programmed to: instruct the controller to adjustthe shutter speed setting of the camera device such that the secondimage is captured with a different shutter speed setting than the firstimage, wherein the adjusting occurs on one of a periodic basis, regularbasis, and random basis.
 5. The system of claim 4, wherein thecontroller is configured to modify an exposure setting of the cameradevice, and wherein the processor is further programmed to: instruct thecontroller to adjust the exposure setting of the camera device such thatthe second image is captured with a different exposure setting than thefirst image, wherein the adjusting of the exposure setting occurs on oneof a periodic basis, regular basis, and random basis.
 6. The system ofclaim 1, further comprising: a controller communicatively connected tothe camera device, wherein the controller is configured to modify a zoomsetting of the camera device; and wherein the processor is furtherprogrammed to: instruct the controller to adjust the zoom setting of thecamera device such that the second image is captured with a differentzoom setting than the first image, wherein the adjusting occurs on oneof a periodic basis, regular basis, and random basis.
 7. The system ofclaim 1, wherein the processor comprises an application-specificintegrated circuit (ASIC) processor, and the camera device iscommunicatively coupled to the processor by a wired connection.
 8. Thesystem of claim 1, wherein the camera device further operates as anenclosure for the processor and the memory arranged therein.
 9. Thesystem of claim 1, wherein the camera device is attached to the lawenforcement vehicle and comprises a plurality of cameras arranged atdifferent locations of the law enforcement vehicle and configured tooperate in a coordinated manner to capture the first image.
 10. Thesystem of claim 9, wherein at least one of the plurality of camerasincludes an unmanned aerial vehicle equipped with video capturecapabilities.
 11. The system of claim 1, wherein the camera device isphysically apart from the processor and is communicatively coupled tothe processor with one of a wired and wireless connection, and whereinthe camera device omits an infrared illumination component.
 12. Thesystem of claim 1, further comprising a location tracking device coupledto the camera device, wherein the processor is further programmed tostamp the first image with a first location of the camera device at thefirst time when the first image is captured by the camera device. 13.The system of claim 1, further comprising a clock, wherein the processoris further programmed to timestamp the first image upon capture by thecamera device.
 14. A method for recognizing a license plate of a targetvehicle, the method comprising steps to: receive, by a processor locatedat a law enforcement vehicle, a first image of the license plate of thetarget vehicle at a first time, wherein the license plate is at a firstposition; receive, by the processor, a second image of the license plateof the target vehicle at a second time, wherein the license plate is ata second position that is different from the first position; align thelicense plate in the first image and the license plate in the secondimage; transform the first image and the second image by geometricallyrectifying to accommodate for relative positions of the license plate atthe first position and the second position, wherein the geometricallyrectifying comprises scaling, warping, or rotating the first image andthe second image; and execute a temporal noise filter on the first imageand the second image to generate a consolidated image, wherein theconsolidated image has a higher probability that characters of thelicense plate are recognized by a computerized optical characterrecognition platform than the license plate in the first image.
 15. Themethod of claim 14, wherein the first image and the second image arecaptured by a camera device at the law enforcement vehicle, the methodcomprising steps to: instruct, by the processor, the camera device toadjust a capture setting of the camera device based on the relativepositions of the license plate at the first position and the secondposition; modify the capture setting of the camera device prior tocapturing a third image of the license plate; capture the third image ofthe license plate; align the license plate in the third image with thelicense plates in the first image and the second image; transform thethird image to geometrically rectify the license plate to accommodatefor relative positions of the license plate; and execute a temporalnoise filter on the third image and the consolidated image to generate anew consolidated image.
 16. The method of claim 14, comprising steps to:detect, by the processor, a first boundary of the license plate in thefirst image; crop, by the processor, the first image to discard outsideof the first boundary of the first image; detect, by the processor, asecond boundary of the license plate in the second image; and crop, bythe processor, the second image to discard outside of the secondboundary of the second image, wherein the processor comprises artificialintelligence for detect operations.
 17. The method of claim 14,comprising steps to: capturing, by a camera device attached to the lawenforcement vehicle, the first image of the license plate, wherein thelicense plate is at the first position when the first image is captured;capturing, by the camera device attached to the law enforcement vehicle,the second image of the license plate, wherein the license plate is atthe second position when the second image is captured; and storing, bythe camera in a memory, the first image and the second image, whereinthe memory is communicatively coupled to the processor.
 18. The methodof claim 14, comprising steps to: receive, by a wireless circuitrycommunicatively coupled to the processor, a command from an externalsystem to cause the processor to activate and capture the first imageand the second image, wherein the external system comprises at least oneof a remote command center, another law enforcement vehicle, and abody-camera device.
 19. A tangible computer-readable medium storingexecutable instructions that, when executed by a processor of a licenseplate recognition system, cause the license plate recognition system to:receive, by the processor, a first image of a license plate of a targetvehicle at a first time, wherein the target vehicle is at a firstposition when the first image is captured by a camera device; receive,by the processor, a second image of the license plate of the targetvehicle at a second time, wherein the target vehicle is at a secondposition that is different from the first position; detect, by theprocessor, a first boundary of the license plate in the first image;detect, by the processor, a second boundary of the license plate in thesecond image; align the license plate in the first image and the licenseplate in the second image; transform the first image and the secondimage to geometrically rectify the license plate to accommodate forrelative positions of the license plate to the camera device, whereinthe license plate is geometrically rectified by scaling, warping, orrotating the first image and the second image; and execute a temporalnoise filter on the first image and the second image to generate aconsolidated image, wherein the consolidated image has a higherprobability that characters of the license plate are recognized by acomputerized optical character recognition platform than the licenseplate in the first image.
 20. The tangible computer-readable medium ofclaim 19, further storing executable instructions that, when executed bythe processor, cause the license plate recognition system to: receive,by a wireless circuitry communicatively coupled to the processor, acommand from an external system that activates the processor andcaptures the first image and the second image using the camera device,wherein the external system comprises at least one of a remote commandcenter, another law enforcement vehicle, and a body-camera device, andwherein the license plate recognition system is attached to the lawenforcement vehicle.